Druid Migration Procedures

I recently was testing a migration of Druid from one environment to completely new and empty infrastructure.

I ported over the deep storage data. I modified the hdfs location bucket info in the druid_segments table before successfully restoring the MySql database.

Upon coordinator startup, I received this error related to druid basic security and metadata base authentication:

Hey Chris,

Do you have a full copy of the stack trace? (It looks cut off.) Is this something that happens consistently or

When you ask “do these settings and passwords have to match” which settings are you referring to and what do you mean by ‘have to match’? It’s fine if two different environments have two different sets of passwords. But also, if you want them to match you should be able to get them to match.

Thanks Gian. We are actually considering eventual migration to imply. I don’t have the full stack trace today but can try to upload it tonight.

When I refer to passwords and settings I’m referring to the druid basic security setup user and passwords. I only mentioned that because the error seemed related to that topic. These two environments are the same version but have different passwords for tls…basic security…MySQL connectivity…etc.

This does not happen consistently…this is the fist time and we are only using the new environment as a poc for druid restore / migration testing.

Hopefully this is enough of the log Gian. Let me know your thoughts please.

coordinator_startup.txt (5.76 KB)

I actually think I figured out the coordinator issue. I truncated data from the druid_config table (as I read the log mentioned there was already data in that table) and coordinator started up fine.
Now I have a new issue after starting up Overlord but I think I know what it is. I had a supervisor running on the source environment with TLS parameters / passwords to consume data from Kafka’s TLS port. The new environment has a different keystore password and I think that is the issue. Now I just need to figure out the best way to handle this issue with the smallest amount of impact to the artifacts that are coming over from the source environment.

Here is the error FYI.

2019-06-01T23:56:52,339 ERROR [KafkaSupervisor-XXXXXXXXXXXX] org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor - Exception starting KafkaSupervisor[XXXXXXXXXX]: {class=org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor, exceptionType=class org.apache.kafka.common.KafkaException, exceptionMessage=Failed to construct kafka consumer}
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:717) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:597) ~[?:?]
at org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor.getKafkaConsumer(KafkaSupervisor.java:1031) ~[?:?]
at org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor.tryInit(KafkaSupervisor.java:945) ~[?:?]
at org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$null$0(KafkaSupervisor.java:383) ~[?:?]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86) ~[java-util-0.13.0-incubating.jar:0.13.0-incubating]
at org.apache.druid.indexing.kafka.supervisor.KafkaSupervisor.lambda$start$2(KafkaSupervisor.java:381) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.io.IOException: keystore password was incorrect
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:44) ~[?:?]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:93) ~[?:?]
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:51) ~[?:?]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:84) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:657) ~[?:?]
… 11 more