Scaling out Tranquility - Increasing ingestion rate

Hi All,

We are evaluating Druid and we are ingesting data using the Tranquility Server.

Currently we use the imply.io bundle and have a Node.JS API which does an HTTP POST request to the Tranquility Server which runs on the Data VM.

http://<DATA_VM>:8200/v1/post/<our_datasource>

Our Druid Cluster has the following specs:

  • Master VM: (Collocating Zookeeper/Overlord/Coordinator/MySQL): 4 cores, 14GB RAM

  • Broker VM: 8 cores, 28GB RAM

  • Data VM: (Collocating: Historical/Middle Manager/Tranquility): 8 cores, 28GB RAM

We provide 3G RAM per peon, we have worker.capacity=6 and in the Tranquility config file we specify segmentGranularity:“FIVE_MINUTE” and task.partitions : “2”

We are dealing with 17 dimensions and 6 metrics.

We have achieved a throughput of 4.5k req/sec by using Apache Benchmark for load generation. Does this throughput sound reasonable for the above cluster specs or should we expect more?

Initially, we had a throughput of 2.8k req/sec and when we increased lingerMillis to 1000 millis, our throughput increased to 4.5k req/sec.

We didn’t fully get why this made such a huge difference. Was it because Tranquility was able to batch requests? Increasing the lingerMillis further to 2000 millis or 5000 millis did not make any difference to throughput. Does anyone know a possible reasoning for this behaviour?

At this stage we would like to increase our ingestion throughput from 4.5k to 20k req/sec.

So we created a second Data VM (Data_VM_2) running Historical/Middle Manager/Tranquility, with the exact same configuration as the DATA_VM_1. Running our load tests showed that throughput remained the same (4.5k req/sec) instead of increasing…

I suppose we are doing something wrong in scaling out Tranquility, so I thought to ask the user group.

Our API still sends events to the Tranquility on our initial Data VM:

http://<DATA_VM_1>:8200/v1/post/<our_datasource>

Is this correct or do we need to put a load balancer in front of both the Data VMs or does Zookeeper automatically knows that there is an extra Tranquility and therefore it is able to ingest more requests per second?

I have attached below our cluster configuration files.

Thanks in advance,

Petros

conf.zip (11.9 KB)

Tranquility understands the number of partitions that you ask for, and will distribute the load evenly.

Segment granularity of five minutes is very small. Hourly is more common.

Above 4k/second I would think that the publishing node app itself would start to become the bottleneck. It depends on how much JSON you’re sending.

Hi Theo, inline.

Hi All,

We are evaluating Druid and we are ingesting data using the Tranquility Server.

https://github.com/druid-io/tranquility/blob/master/docs/server.md

Currently we use the imply.io bundle and have a Node.JS API which does an HTTP POST request to the Tranquility Server which runs on the Data VM.

http://<DATA_VM>:8200/v1/post/<our_datasource>

Our Druid Cluster has the following specs:

  • Master VM: (Collocating Zookeeper/Overlord/Coordinator/MySQL): 4 cores, 14GB RAM
  • Broker VM: 8 cores, 28GB RAM
  • Data VM: (Collocating: Historical/Middle Manager/Tranquility): 8 cores, 28GB RAM

We provide 3G RAM per peon, we have worker.capacity=6 and in the Tranquility config file we specify segmentGranularity:“FIVE_MINUTE” and task.partitions : “2”

We are dealing with 17 dimensions and 6 metrics.

We have achieved a throughput of 4.5k req/sec by using Apache Benchmark for load generation. Does this throughput sound reasonable for the above cluster specs or should we expect more?

Initially, we had a throughput of 2.8k req/sec and when we increased lingerMillis to 1000 millis, our throughput increased to 4.5k req/sec.

This is much slower than we normally benchmark.

We didn’t fully get why this made such a huge difference. Was it because Tranquility was able to batch requests?

Tranquility doesn’t batch requests, it is very much a true streaming system.

Increasing the lingerMillis further to 2000 millis or 5000 millis did not make any difference to throughput. Does anyone know a possible reasoning for this behaviour?

At this stage we would like to increase our ingestion throughput from 4.5k to 20k req/sec.

This sounds more reasonable about Tranq performance.

So we created a second Data VM (Data_VM_2) running Historical/Middle Manager/Tranquility, with the exact same configuration as the DATA_VM_1. Running our load tests showed that throughput remained the same (4.5k req/sec) instead of increasing…

I suppose we are doing something wrong in scaling out Tranquility, so I thought to ask the user group.

Gian (creator) is probably the best person to look into this further.

Hey Petros,

How many messages are you sending per HTTP request and how many concurrent connections do you have from your load tester?

If it’s a small number of messages, the biggest thing you can do to boost throughput is include more messages per HTTP request. Aiming for a MB or two per POST body is usually good. Also opening more concurrent connections can help, as that gets the pipeline flowing better.

And yeah, if you add more Tranquility Servers, you should talk to them via a load balancer.

Thank you guys for the great responses!

@Max: We are currently using 5 minute segment granularity only for testing purposes. In production we will be using HOUR segment granularity.

Gian’s suggestion worked fine! It seemed that the bottleneck was opening and closing the TCP connections because we were sending 1 event per request.

I am presenting our findings here, so that it might help other users.

  • First step was to batch 100 events per HTTP request when sending to Tranquility. This increased throughput to from 4.5k req/sec to 25k req/sec.

  • Then we put the 2 Tranquility instances behind a load balancer, which increased throughput to 55k req/sec.

We get this throughput with concurrency level 150.

Great to hear!

Gian,

I have a similar question, so thought of high jacking this thread.

Apart from writing a separate http client, what are the pros and cons of using Tranquility Core vs Tranquility Server. Which one gives better through put?

The throughput should be similar with Core vs Server assuming both are being used in the highest performing way. For Core this means avoiding blocking on futures too much, for Server this means using good post batch sizes and using a good number of connections.

The one downside of Server is that it needs an extra hop (your program -> Tranquility Server -> Druid) rather than sending directly from your program to Druid. This incurs some extra network traffic and serde overhead.