java.lang.OutOfMemoryError: Direct buffer memory : KAFKA Indexing Service

java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Direct buffer memory

at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-16.0.1.jar:-?]

at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-16.0.1.jar:-?]

at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-16.0.1.jar:-?]

at
io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver.publishAll(FiniteAppenderatorDriver.java:423)
[druid-server-0.10.0.jar:0.10.0]

Attaching our runtime.properties and jvm.properties. We have not defined -XX:MaxDirectMemorySize in jvm.properties.

Surprising thing is same configuration was working earlier and there is no change to the data or the size and it has started breaking kafka indexing tasks now.

Also i wanted to know what is the best way we can measure how much druid.processing.buffer.sizeBytes/druid.processing.numThreads required for our data ? What this each thread is responsible for - we can keep buffer size for that thread accordingly.

Regards,

Arpan Khagram

+91 8308993200

jvm.config (228 Bytes)

runtime.properties (948 Bytes)