I recently stood up our Druid cluster in a Rancher environment. Right now, I simply mount the host volume into the containers when submitted. I had a few questions as well as hoped for some advice when it comes to things such as data persistence. Ideally, I can hot mount an existing EBS volume to a new instance that comes up. That is a possibility as well as using a number of NFS / EBS options (AWS cloud based stack here). I chose the hot mount approach as a third party software we use provides this functionality for our EC2 instances out of the box.
I’m wondering if there is a way to take a new historical (spun up on the new EC2 instance in the case our cluster scales), and hot mount that EBS volume into the new historical node’s running container. Essentially, I want to know if it can share-data and if the data can pick up where the previous historical left off. That way if I lose an EC2 instance, but still have its data volume, and I spin up a new EC2 instance with the old instance’s data volume would it essentially persist the working data and the new containers that are automatically run on the new EC2 host would just continue along happily. Make sense?
Second question - what types of conflicts would arise from mounting the same volume path into two historical containers running on the same host. Essentially, they would be sharing the same mount. Is there built-in functionality that would let them distinguish between each other, or a way to prevent conflicts (configuration in runtime.properties, etc…)?
Thank you very much for any assistance!