Horizontal Scaling?

I read every page on druid.io , and forgive me if I forgot, but I didn’t see anywhere that mentions horizontal scalability, except the part regarding redundancy I pasted here - that the other server would pick up if the main one died.

High Availability Characteristics

Druid is designed to have no single point of failure. Different node types are able to fail without impacting the services of the other node types. To run a highly available Druid cluster, you should have at least 2 nodes of every node type running.

If I want my historical node to hold more data - is the only option vertical scaling? i.e. making the instance beefier with more CPU / RAM / SSD

Thanks for any answers,

Geoff

Hey Geoff,

You can start as many historical nodes as you like. Specifically the scalability and availability works like this,

  1. Historicals, middleManagers, brokers, and realtime nodes scale horizontally. Historicals and middleManagers also support replication of segments and tasks (respectively) for high availability of data.

  2. Overlords and coordinators scale vertically. You can deploy more than one for failover. They both deal with metadata, not your actual data, and neither are part of the query path.

Gian,

Thanks for reply. A follow-up:

How does Historicals support replication of segments? E.g. if we have 3 instances, would 1.5 instances’ worth of resources be used to contain segments so that the other 1.5 instances’ worth of resources can be used to hold the replicates for high availability?

And internally historicals make sure the main segment and the replicated segment are on different instances?

I’m asking to see how spinning up new instances can help scaling. If my above questions are indeed the case, 10 instances would only mean 5 instances worth of capacity. Our use case actually doesn’t require high availability - so if we can somehow opt out of the replication that would be even better.

Geoff

Yep, that’s how the replication works. The system does make sure that the replicas are on different servers.

You can control the number of replicas. By default it’s 2 but you could set it to 1; in that case, any time a historical node goes down, some segments will be unavailable.

Thanks much for further clarification. Hopefully a final follow-up:

From what I understand for batch data, all data are available from s3, and are loaded to historical nodes to be served - so if we have 2 historical nodes, and a query needs both nodes for the data and 1 node goes down, shouldn’t druid know to load back the data lost in 1 node from s3 to the remaining node?

Sure, if the query spans a lot of data, this may need some disk access as memory/memory-mapped in the remaining node are not enough, but correctness should not necessarily have to be compromised, right? Maybe this is a TODO? Or is there another general way to handle this situation?

Druid will load unavailable or under-replicated data from S3, but this happens in the background, not on-demand in response to queries. So while it’s loading, queries results will not reflect those segments that are still being loaded back up.

I see. thanks.
BTW, I can’t find the replication property to set. I searched the downloaded, and the website. What’s the property name, and should I set it in the historical runtime.properties file?

The replication count is set in the coordinator console: http://coordinator_ip:port

Click the pencil in the top left.