We are evaluating Druid and we are ingesting data using the Tranquility Server.
Currently we use the imply.io bundle and have a Node.JS API which does an HTTP POST request to the Tranquility Server which runs on the Data VM.
Our Druid Cluster has the following specs:
Master VM: (Collocating Zookeeper/Overlord/Coordinator/MySQL): 4 cores, 14GB RAM
Broker VM: 8 cores, 28GB RAM
Data VM: (Collocating: Historical/Middle Manager/Tranquility): 8 cores, 28GB RAM
We provide 3G RAM per peon, we have worker.capacity=6 and in the Tranquility config file we specify segmentGranularity:“FIVE_MINUTE” and task.partitions : “2”
We are dealing with 17 dimensions and 6 metrics.
We have achieved a throughput of 4.5k req/sec by using Apache Benchmark for load generation. Does this throughput sound reasonable for the above cluster specs or should we expect more?
Initially, we had a throughput of 2.8k req/sec and when we increased lingerMillis to 1000 millis, our throughput increased to 4.5k req/sec.
We didn’t fully get why this made such a huge difference. Was it because Tranquility was able to batch requests? Increasing the lingerMillis further to 2000 millis or 5000 millis did not make any difference to throughput. Does anyone know a possible reasoning for this behaviour?
At this stage we would like to increase our ingestion throughput from 4.5k to 20k req/sec.
So we created a second Data VM (Data_VM_2) running Historical/Middle Manager/Tranquility, with the exact same configuration as the DATA_VM_1. Running our load tests showed that throughput remained the same (4.5k req/sec) instead of increasing…
I suppose we are doing something wrong in scaling out Tranquility, so I thought to ask the user group.
Our API still sends events to the Tranquility on our initial Data VM:
Is this correct or do we need to put a load balancer in front of both the Data VMs or does Zookeeper automatically knows that there is an extra Tranquility and therefore it is able to ingest more requests per second?
I have attached below our cluster configuration files.
Thanks in advance,
conf.zip (11.9 KB)