we’re currently looking at implementing a Druid.io cluster.
What would happen if we would run the historical and broker nodes on ephemeral disks.
Does druid care if i kill a broker or a historical node and spin another new one up?
Are you storing your segments in deep storage as recommended in HDFS or in Cassandra?
At a high level, yes, historical and broker nodes can come and go. You just have to provision enough nodes to ensure HA. For example, you’ll want to utilize segment replication so that any given segment is cached on more than one historical node, and you’ll also want your query clients to be able to handle broker connection failures by either getting broker information from zookeeper and/or a known broker list and performing failover. The historical node and coordinator node documentation pages give hints on how segments are delegated to historical nodes:
http://druid.io/docs/0.10.1/design/coordinator.html (Segment Availability)
Your brokers can also use shared distributed caching (memcached) so that your cached results don’t get clobbered if a broker borks