Index task ignore druid.indexer.task.baseDir

Hi guys
I`m trying to run batch index hadoop task, ingesting file from hdfs and storing output in hdfs

my task fails all the time. in logs I noticed that extensions are loaded multiple times when starting the task, which causes the task to fail with
Caused by: java.lang.UnsatisfiedLinkError: Native Library /tmp/mapr-druid-libMapRClient.5.1.0-mapr.so already loaded in another classloader

``

this is because druid-hdfs-storage is loaded multiple times
mvince@dwh-dev-druid-m-01:/data/druid/log$ cat index_hadoop_kpi_registrations_experimental_2017-02-28T11:50:34.963Z.log | grep “Loading extension [druid-hdfs-storage]”
2017-02-28T12:50:36,920 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [io.druid.cli.CliCommandCreator]
2017-02-28T12:50:37,134 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [io.druid.initialization.DruidModule]
2017-02-28T12:50:42,287 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [io.druid.initialization.DruidModule]

``

I tried to set druid.indexer.task.baseDir in my middle manager settings and also pass -Djava.io.tmpdir parameter in druid.indexer.runner.javaOptsArray in middle manager settings, but i can see this is ignored in index task log:


2017-02-28T12:50:39,891 INFO [main] io.druid.cli.CliPeon - * druid.indexer.runner.javaOpts: -Djava.io.tmpdir=/data/druid/tmp22 -server -Xmx3072m -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Duser.time
zone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Djava.io.tmpdir=/data/druid/tmp2
2017-02-28T12:50:39,891 INFO [main] io.druid.cli.CliPeon - * druid.indexer.runner.javaOptsArray: ["-Djava.io.tmpdir=/data/druid/tmp4"]
2017-02-28T12:50:39,891 INFO [main] io.druid.cli.CliPeon - * druid.indexer.task.baseDir: /data/druid/tmp3
2017-02-28T12:50:39,891 INFO [main] io.druid.cli.CliPeon - * druid.indexer.task.baseTaskDir: /data/druid/task

2017-02-28T12:50:39,894 INFO [main] io.druid.cli.CliPeon - * java.io.tmpdir: /tmp

``

in middle manager log I can see that peon was started with multiple -Djava.io.tmpdir parameters

2017-02-28T12:50:35,503 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Running command: java -cp conf/druid/_common:conf/druid/middleManager:lib/base64-2.3.8.jar:lib/aws-java-sdk-importexport-1.10.21.jar:lib/netty-3.10.4.Final.jar:lib/aws-java-sdk-cloudformation-1.10.21.jar:lib/aws-java-sdk-s3-1.10.21.jar:lib/protobuf-java-2.5.0.jar:lib/curator-client-2.11.0.jar:lib/jcl-over-slf4j-1.7.12.jar:lib/commons-dbcp2-2.0.1.jar:lib/commons-cli-1.2.jar:lib/disruptor-3.3.0.jar:lib/slf4j-api-1.7.12.jar:lib/zookeeper-3.4.9.jar:lib/aws-java-sdk-cloudhsm-1.10.21.jar:lib/druid-processing-0.9.2.jar:lib/jetty-security-9.2.5.v20141112.jar:lib/aether-impl-0.9.0.M2.jar:lib/geoip2-0.4.0.jar:lib/jets3t-0.9.4.jar:lib/druid-indexing-service-0.9.2.jar:lib/log4j-jul-2.5.jar:lib/aws-java-sdk-opsworks-1.10.21.jar:lib/aws-java-sdk-cognitosync-1.10.21.jar:lib/jsr311-api-1.1.1.jar:lib/curator-x-discovery-2.11.0.jar:lib/log4j-core-2.5.jar:lib/maven-model-builder-3.1.1.jar:lib/jetty-server-9.2.5.v20141112.jar:lib/joda-time-2.8.2.jar:lib/guice-multibindings-4.1.0.jar:lib/derbyclient-10.11.1.1.jar:lib/aws-java-sdk-datapipeline-1.10.21.jar:lib/aether-util-0.9.0.M2.jar:lib/jackson-datatype-guava-2.4.6.jar:lib/bcprov-jdk15on-1.52.jar:lib/jackson-databind-2.4.6.jar:lib/okhttp-1.0.2.jar:lib/aws-java-sdk-logs-1.10.21.jar:lib/hibernate-validator-5.1.3.Final.jar:lib/commons-lang-2.6.jar:lib/druid-indexing-hadoop-0.9.2.jar:lib/aether-spi-0.9.0.M2.jar:lib/mapdb-1.0.8.jar:lib/httpcore-4.4.3.jar:lib/aws-java-sdk-ses-1.10.21.jar:lib/spymemcached-2.11.7.jar:lib/lz4-1.3.0.jar:lib/aws-java-sdk-kinesis-1.10.21.jar:lib/aws-java-sdk-1.10.21.jar:lib/aws-java-sdk-elasticloadbalancing-1.10.21.jar:lib/aws-java-sdk-cloudwatchmetrics-1.10.21.jar:lib/jetty-io-9.2.5.v20141112.jar:lib/curator-framework-2.11.0.jar:lib/maven-settings-builder-3.1.1.jar:lib/aws-java-sdk-config-1.10.21.jar:lib/jackson-core-asl-1.9.13.jar:lib/jackson-dataformat-smile-2.4.6.jar:lib/aws-java-sdk-codecommit-1.10.21.jar:lib/aws-java-sdk-devicefarm-1.10.21.jar:lib/validation-api-1.1.0.Final.jar:lib/maxminddb-0.2.0.jar:lib/javax.inject-1.jar:lib/guice-4.1.0.jar:lib/jline-0.9.94.jar:lib/extendedset-1.3.10.jar:lib/jsr305-2.0.1.jar:lib/classmate-1.0.0.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/aws-java-sdk-workspaces-1.10.21.jar:lib/curator-recipes-2.11.0.jar:lib/jetty-continuation-9.2.5.v20141112.jar:lib/log4j-slf4j-impl-2.5.jar:lib/config-magic-0.9.jar:lib/aws-java-sdk-emr-1.10.21.jar:lib/jersey-guice-1.19.jar:lib/aws-java-sdk-storagegateway-1.10.21.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/druid-common-0.9.2.jar:lib/guava-16.0.1.jar:lib/icu4j-4.8.1.jar:lib/jackson-core-2.4.6.jar:lib/aws-java-sdk-simpledb-1.10.21.jar:lib/aws-java-sdk-efs-1.10.21.jar:lib/aws-java-sdk-directory-1.10.21.jar:lib/aws-java-sdk-codepipeline-1.10.21.jar:lib/aws-java-sdk-elasticache-1.10.21.jar:lib/aws-java-sdk-cloudsearch-1.10.21.jar:lib/aws-java-sdk-ec2-1.10.21.jar:lib/log4j-1.2-api-2.5.jar:lib/java-xmlbuilder-1.1.jar:lib/jersey-servlet-1.19.jar:lib/http-client-1.0.4.jar:lib/javax.el-3.0.0.jar:lib/activation-1.1.1.jar:lib/jetty-servlets-9.2.5.v20141112.jar:lib/aws-java-sdk-route53-1.10.21.jar:lib/commons-pool-1.6.jar:lib/aws-java-sdk-kms-1.10.21.jar:lib/maven-repository-metadata-3.1.1.jar:lib/bytebuffer-collections-0.2.5.jar:lib/compress-lzf-1.0.3.jar:lib/druid-aws-common-0.9.2.jar:lib/annotations-2.0.3.jar:lib/aws-java-sdk-sns-1.10.21.jar:lib/commons-codec-1.7.jar:lib/aws-java-sdk-cloudfront-1.10.21.jar:lib/aws-java-sdk-lambda-1.10.21.jar:lib/javax.el-api-3.0.0.jar:lib/commons-math3-3.6.1.jar:lib/maven-aether-provider-3.1.1.jar:lib/jetty-proxy-9.2.5.v20141112.jar:lib/java-util-0.27.10.jar:lib/jersey-core-1.19.jar:lib/irc-api-1.0-0014.jar:lib/jboss-logging-3.1.3.GA.jar:lib/aws-java-sdk-sts-1.10.21.jar:lib/commons-logging-1.1.1.jar:lib/wagon-provider-api-2.4.jar:lib/aws-java-sdk-ssm-1.10.21.jar:lib/jetty-client-9.2.5.v20141112.jar:lib/jdbi-2.63.1.jar:lib/plexus-interpolation-1.19.jar:lib/aws-java-sdk-core-1.10.21.jar:lib/aws-java-sdk-elasticbeanstalk-1.10.21.jar:lib/plexus-utils-3.0.15.jar:lib/jackson-module-jaxb-annotations-2.4.6.jar:lib/javax.servlet-api-3.1.0.jar:lib/log4j-api-2.5.jar:lib/guice-servlet-4.1.0.jar:lib/jetty-servlet-9.2.5.v20141112.jar:lib/aws-java-sdk-glacier-1.10.21.jar:lib/jackson-jaxrs-base-2.4.6.jar:lib/aws-java-sdk-cloudtrail-1.10.21.jar:lib/jackson-datatype-joda-2.4.6.jar:lib/emitter-0.3.6.jar:lib/druid-api-0.9.2.jar:lib/maven-model-3.1.1.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/maven-settings-3.1.1.jar:lib/json-path-2.1.0.jar:lib/aws-java-sdk-swf-libraries-1.10.21.jar:lib/server-metrics-0.2.8.jar:lib/commons-pool2-2.2.jar:lib/airline-0.7.jar:lib/aws-java-sdk-ecs-1.10.21.jar:lib/aws-java-sdk-machinelearning-1.10.21.jar:lib/aws-java-sdk-rds-1.10.21.jar:lib/derby-10.11.1.1.jar:lib/jackson-jaxrs-json-provider-2.4.6.jar:lib/druid-services-0.9.2.jar:lib/aws-java-sdk-sqs-1.10.21.jar:lib/aws-java-sdk-support-1.10.21.jar:lib/jetty-util-9.2.5.v20141112.jar:lib/rhino-1.7R5.jar:lib/aws-java-sdk-autoscaling-1.10.21.jar:lib/aws-java-sdk-iam-1.10.21.jar:lib/aws-java-sdk-dynamodb-1.10.21.jar:lib/commons-io-2.4.jar:lib/httpclient-4.5.1.jar:lib/aether-api-0.9.0.M2.jar:lib/jackson-jaxrs-smile-provider-2.4.6.jar:lib/aws-java-sdk-codedeploy-1.10.21.jar:lib/derbynet-10.11.1.1.jar:lib/jersey-server-1.19.jar:lib/jackson-annotations-2.4.6.jar:lib/aws-java-sdk-directconnect-1.10.21.jar:lib/opencsv-2.3.jar:lib/antlr4-runtime-4.5.1.jar:lib/aws-java-sdk-redshift-1.10.21.jar:lib/aws-java-sdk-simpleworkflow-1.10.21.jar:lib/jetty-http-9.2.5.v20141112.jar:lib/druid-server-0.9.2.jar:lib/RoaringBitmap-0.5.18.jar:lib/tesla-aether-0.0.5.jar:lib/aws-java-sdk-cloudwatch-1.10.21.jar:lib/aws-java-sdk-cognitoidentity-1.10.21.jar:lib/aws-java-sdk-elastictranscoder-1.10.21.jar:lib/aopalliance-1.0.jar:lib/google-http-client-jackson2-1.15.0-rc.jar:lib/druid-console-0.0.2.jar:/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/*
-Djava.io.tmpdir=/data/druid/tmp22 -server -Xmx3072m -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Djava.io.tmpdir=/data/druid/tmp2
-Djava.io.tmpdir=/data/druid/tmp4
-Ddruid.indexer.task.baseTaskDir=/data/druid/task
-Ddruid.host=****
-Ddruid.discovery.curator.path=/dev/discovery
-Ddruid.processing.buffer.sizeBytes=536870912
-Ddruid.metadata.storage.connector.password=****
-Duser.timezone=UTC
-Dfile.encoding.pkg=sun.io
-Ddruid.storage.storageDirectory=/tmp/druid/persistent
-Ddruid.selectors.coordinator.serviceName=druid/coordinator
-Ddruid.selectors.indexing.serviceName=druid/overlord
-Ddruid.indexer.task.baseDir=/data/druid/tmp3
-Ddruid.port=8091
-Ddruid.server.http.numThreads=25
-Ddruid.worker.capacity=1
-Ddruid.service=druid/middleManager
-Ddruid.metadata.storage.connector.user=****
-Ddruid.metadata.storage.type=mysql
-Ddruid.metadata.storage.connector.connectURI=****
-Djava.io.tmpdir=/tmp
-Ddruid.extensions.loadList=[“druid-kafka-eight”, “mysql-metadata-storage”, “druid-distinctcount”, “druid-kafka-indexing-service”, “druid-stats”, “druid-hdfs-storage”]
-Ddruid.startup.logging.logProperties=true
-Ddruid.processing.numThreads=2
-Ddruid.zk.service.host=*****
-Ddruid.indexer.logs.directory=/data/druid/log
-Ddruid.zk.paths.base=/druid/dev
-Dfile.encoding=UTF-8
-Ddruid.storage.type=local
-Ddruid.worker.ip=****
-Ddruid.indexer.logs.type=file
-Ddruid.metrics.emitter.dimension.dataSource=kpi_registrations_experimental
-Ddruid.metrics.emitter.dimension.taskId=index_hadoop_kpi_registrations_experimental_2017-02-28T11:50:34.963Z
-Ddruid.host=****
-Ddruid.port=8100 io.druid.cli.Main internal peon /data/druid/task/index_hadoop_kpi_registrations_experimental_2017-02-28T11:50:34.963Z/task.json /data/druid/task/index_hadoop_kpi_registrations_experimental_2017-02-28T11:50:34.963Z/cace2d7f-ddd2-4353-b49d-b3db405e2075/status.json

``

middle manager conf with command how i start druid node is attached

middlemanager.conf (2.08 KB)

And I`m running druid 0.9.2, MapR hadoop cluster (hadoop 2.7.0)