Druid's Maximum Throughput

Hello All,

Please answer the following questions:

1.What is maximum throughput for data ingestion in druid for real time data?.

2.What is the maximum throughput for batch ingestion.

Consider we are using standard hardware.

Thanks,

Ashish

Hey Ashish,

In theory there is no limit, since Druid ingestion uses minimal locking and is linearly scalable.

In practice once you reach a few Million QPS you are going to encounter a bunch of cost pressure to make sure you are optimizing things, and need to have a solid operational base to handle the one-off errors that pop up randomly. You’ll also need a good relationship with your cloud provider to make sure you don’t get frozen out on resources in a particular zone you operate in. So the druid part tends to work, but there are a lot of other business needs at that point.

Query load also matters a lot here as well since the real-time part is the part that doesn’t have a lot of effective levers to scale up with higher query load. If the total system is taking in millions of events per second and serving hundreds of ad-hoc queries per second, then you’re running into a risky territory without some special design prior to shipping the system.

Thanks Charles for the “in practice” answer rather than my “in theory” answer :slight_smile:

Query load also matters a lot here as well since the real-time part is the part that doesn’t have a lot of effective levers to scale up with higher query load.

By the way, there’s a proposal out that should help a ton with this: https://github.com/apache/incubator-druid/issues/7900