Ingestion Rate Setting in Community Druid

Hi,

We are using Community Druid to ingest from kafka topics. Where do I check the ingestion rate setting in Druid?

Hello!

Happy holidays. There is really no concept of ingestion rate “setting” in druid. It will pull data as fast is it can based on the parameters set in the supervisor spec.

Are you trying to increase the ingestion rate? Or limit it?

Thanks,
Vijeth

It is more of to help ease with the OOM issues being encountered despite adjusting the mem capacity and heap size already.

I think we found the setting that helps addresses it - maxRowsInMemory. Lowered this to half of the default for a high volume task so it writes to disk more often.

Curious though what is the tradeoff of lowering this setting.

1 Like

Hi @magnusarcher,

To answer your question as simply as I can, the tradeoff is your ingestion rate may be slower as the task has to pause to persist the data more often.

There are ways to remedy this if your ingestion lag becomes unacceptable.

And thank you for updating us with your solution!

Thanks,
Vijeth_Sagar