Throughput - 1 Million saves per second
Hardware -
16 vCPU
48GB RAM
800GB SSD
Druid Data Node Configs-
MIDDLE MANAGER CONFIGS
druid.service=druid/middleManager
druid.plaintextPort=8091
Number of tasks per middleManager
druid.worker.capacity=15
Task launch parameters
druid.indexer.runner.javaOpts=-server -Xms256m -Xmx3g -XX:MaxDirectMemorySize=10g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task
HTTP server threads
druid.server.http.numThreads=50
Processing threads and buffers on Peons
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100000000
druid.indexer.fork.property.druid.processing.numThreads=1
Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
HISTORICAL CONFIGS
druid.service=druid/historical
druid.plaintextPort=8083
HTTP server threads
druid.server.http.numThreads=60
Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=60
druid.processing.numThreads=1
druid.processing.tmpDir=var/druid/processing
druid.broker.http.numConnections=25
Segment storage
druid.segmentCache.locations=[{“path”:“var/druid/segment-cache”,“maxSize”:300000000000}]
druid.server.maxSize=300000000000
Query cache
druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=256000000
Issue–
RAM runs out with 1 hour of starting the load test. Testing with 1 - 1.5 Million saves per second
3 Zookeepers
3 Kafkas
3 Druid Data Nodes, 16 Tasks
Buffer is filled up by the mmaps.