How to increase ingestion performance on Tranquility with Kafka

Hi,

I am new to tranquility with kafka, and I am trying to increase ingestion performance.

Druid : 0.9.0-rc1

Tranquility : 0.7.3

Kafka(0.8.2) topics : 30

ingestion spec

In my case, It seems that the first bottleneck is the tranquility kafka throughput, not druid ingestion.

I need to ingest 200k event/sec (120000000/10minute).

One tranquility kafka is deployed, it received 1620k/ 15secs, (108k/sec, 6480000/10minute).

I revised the kafka consumer config, but it did not effect the performance.

So I deployed another instance of tranquility kafka on the different machine to scale out.

Partitions are assigned to 2 tranquility instances gracefully.

However the throughput get worst.

Each tranquility kafka received 550k/15secs, sum of two instance is 1100k/15secs(74k/sec, 44000000/10minute).

I am not sure If I deploy the multiple tranquility instances in the right way and the multiple tranquility instances can increase the throughput.

Followings are the config of tranquility kafka and two instances use the same config.

Please let me know if the way is wrong, and provide some ideas to increase the tranquility throughput.

Kafka topics : 50

------- Configs ----------------

“ioConfig” : {

      "type" : "realtime"

    },

    "tuningConfig" : {

      "type" : "realtime",

      "maxRowsInMemory" : "500000",

      "intermediatePersistPeriod" : "PT7M",

      "windowPeriod" : "PT5M"

    }

  },

  "properties" : {

    "task.partitions" : "1",

    "task.replicants" : "1",

    "topicPattern" : "sip",

    "task.warmingPeriod" : "PT1M",

    "druidBeam.firehoseBufferSize" : "200000",

    "druidBeam.firehoseChunkSize" : "1000"

  }

}

},

“properties” : {

...

"commit.periodMillis" : "15000",

"consumer.numThreads" : "25",

...    

“kafka.group.id” : “tranquility-kafka-cluster-10-1”,

"kafka.num.consumer.fetchers" : "1",

"kafka.fetch.wait.max.ms" : "10"

}

Hey Azrael,

Could you try upgrading to tranquility-0.7.4 if you haven’t already done that? It has a number of performance improvements.

You can also try:

  • raising your commit.periodMillis. Each commit requires a flush and there is some overhead to doing that.

  • raising your tranquility.maxBatchSize, this might get you better throughput.

Thanks Gian, It’s very helpful.

Now, I am trying tranquility-0.7.4, it has definitely performance improvements.

I raise the tranquility instances up to 3 and partition = 3 for the parallelism.

This covers the 200k/sec ingestion throughput.

And the following conversation is very helpful:

https://groups.google.com/forum/m/#!topic/druid-development/_y_lb3t1IZg

2016년 3월 22일 화요일 오전 3시 52분 54초 UTC+9, Gian Merlino 님의 말: