Error starting CliPeon when streaming data via Tranquility Kafka

Hi all,

I’ve recently started using Druid (so far really impressed with the technology!) and I’ve been experimenting with streaming avro encoded data from Kafka using Tranquility Kafka.

When a new realtime task is created I see this error in the log:

2017-05-05T15:36:52,514 ERROR [main] io.druid.cli.CliPeon - Error when starting up.  Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, druid.processing.numThreads, or druid.processing.numMergeBuffers: maxDirectMemory[1,908,932,608], memoryNeeded[2,147,483,648] = druid.processing.buffer.sizeBytes[536,870,912] * (druid.processing.numMergeBuffers[2] + druid.processing.numThreads[1] + 1)


I have already tried updating the MiddleManager -XX:MaxDirectMemorySize to 30gb in the appropriate jvm.properties files. However, it doesn't appear that this is working. Is this the correct place to update the memory setting for this error? Previous older threads suggested this is the case, however, I'm seeing this error repeatedly, even after updating the config to multiple different values.

ps -ef | grep java | grep middleManager
root     22713 15312  0 15:14 pts/2    00:00:12 java -server -Xms64m -Xmx64m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.io.tmpdir=var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -XX:MaxDirectMemorySize=30g -cp conf/druid/_common:conf/druid/middleManager:lib/* io.druid.cli.Main server middleManager


Here’s some setup details:

I have a test cluster with overlord x1, coordinator x1, middleManager x3, broker x3, historical x3

The versions I’m using are:

Tranquility 0.8.2

Druid 0.10.0

druid-avro-extensions 0.10.0 [Using Confluent Schema Registry]

Appreciate any help!

Thanks,

Tim

The error message you’re seeing is for the task, so you shouldn’t add the config to the middleManager jvm.config, it should be in its runtime.properties under druid.indexer.runner.javaOpts. Wheels within wheels.

That being said, you could probably do with a somewhat smaller druid.processing.buffer.sizeBytes. 500MB is usually overkill for realtime tasks since they generally don’t deal with as much data as historical nodes. I usually do 200MB.

Thanks for the prompt response Gian,
I had just noticed there was some reference to this in http://druid.io/docs/latest/configuration/production-cluster.html - so I’m trying out both reducing the buffer size and also increasing the MaxDirectMemorySize.

It seems that has helped me past that particular error!!

Thanks,

Tim