Upgrading Druid from 0.9.2 to 0.10.0, Ingestion task is failing because of kerberos authentication

Hi,

I am upgrading my druid cluster from 0.9.2 to 0.10.0.I have build druid with 2.7.1 as my hadoop cluster is on 2.7.1.

In my cluster I am using kerberos authentication which was configured as below for druid 0.9.2 :

druid.storage.type=hdfs

druid.storage.storageDirectory=hdfs://dev16ha:8020/some_dir

druid.hadoop.security.kerberos.principal=principal

druid.hadoop.security.kerberos.keytab=keytab path

And this configuration was working fine.

But when I upgraded to 0.10.0 , it gave below exception

Error injecting constructor, java.lang.IllegalArgumentException: Wrong FS: hdfs://dev16ha:8020/some_dir, expected: hdfs://dev16ha

at io.druid.storage.hdfs.HdfsDataSegmentPusher.(HdfsDataSegmentPusher.java:61)

while locating io.druid.storage.hdfs.HdfsDataSegmentPusher

at io.druid.storage.hdfs.HdfsStorageDruidModule.configure(HdfsStorageDruidModule.java:97) (via modules: com.google.inject.util.Modules$OverrideModule -> io.druid.storage.hdfs.HdfsStorageDruidModule)

while locating io.druid.segment.loading.DataSegmentPusher annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=101, type=MAPBINDER, keyType=java.lang.String)

at io.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:86) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.guice.LocalDataStorageDruidModule)

while locating io.druid.segment.loading.DataSegmentPusher

for the 4th parameter of io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:94)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:155) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliPeon$1)

while locating io.druid.indexing.common.TaskToolboxFactory

for the 1st parameter of io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:96)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:182) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliPeon$1)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.indexing.overlord.TaskRunner

for the 4th parameter of io.druid.indexing.worker.executor.ExecutorLifecycle.<init>(ExecutorLifecycle.java:79)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:172) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> io.druid.cli.CliPeon$1)

while locating io.druid.indexing.worker.executor.ExecutorLifecycle

When I removed the port other exception which came :

Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: yarn/tstr400189@HADOOP_DEV16.Test.COM

2017-04-27T09:39:09,303 INFO [task-runner-0-priority-0] org.apache.hadoop.io.retry.RetryInvocationHandler - Exception while invoking getNewApplication of class ApplicationClientProtocolPBClientImpl over rm2 after 1 fail over attempts. Trying to fail over immediately.

java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: yarn/tstr400189@HADOOP_DEV16.Test.COM; Host Details : local host is: “Abacus-Dev-Druid-200636141-5-214521906/10.117.138.249”; destination host is: “tstr400189”:8032;

I am trying to resolve this meanwhile if anyone have encountered the same problem please let me know how to resolve.

am I missing any configuration here.

I think that means your hadoop configuration (maybe from XMLs) references hdfs://dev16ha (no port) but your Druid config has a port in it. Try making them match and see if that helps.