[druid-user] increasing no of waiting tasks for parallel batch ingestion

Hi all,

We are working on a druid cluster with 3 master nodes,2 data nodes and 2 query nodes.We are having an issue when injecting data into the cluster that tasks status keep hanging long time to move from waiting for the stage to success.I have attache a screenshot and configs of nodes. Please assist to reslove this.

middlemanager- jvm

-server
-Xms128m
-Xmx128m
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC+5:30
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

middlemanager-runtime

druid.service=druid/middleManager
druid.plaintextPort=8091

Number of tasks per middleManager

druid.worker.capacity=4

Task launch parameters

druid.indexer.runner.javaOpts=-server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC+05:30 -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task

HTTP server threads

druid.server.http.numThreads=60

Processing threads and buffers on Peons

druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=100MiB
druid.indexer.fork.property.druid.processing.numThreads=1

Hadoop indexing

historical jvm

-server
-Xms30g
-Xmx30g
-XX:MaxDirectMemorySize=13g
-XX:+ExitOnOutOfMemoryError
-Duser.timezone=UTC+05:30
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

historical runtime

druid.service=druid/historical
druid.plaintextPort=8083

HTTP server threads

druid.server.http.numThreads=60

Processing threads and buffers

druid.processing.buffer.sizeBytes=500MiB
druid.processing.numMergeBuffers=4
druid.processing.numThreads=15
druid.processing.tmpDir=var/druid/processing

Segment storage

druid.segmentCache.locations=[{“path”:"/data/druid/segment-cache",“maxSize”:“27487790694400”}]

Query cache

druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=256MiB

coordinator overload- jvm

-server
-Xms15g
-Xmx15g
-XX:+ExitOnOutOfMemoryError
-XX:+UseG1GC
-Duser.timezone=UTC+05:30
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
-Dderby.stream.error.file=var/druid/derby.log

cordinator-overload- runtime

druid.service=druid/coordinator
druid.plaintextPort=8081

druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S

Run the overlord service in the coordinator process

druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord

druid.indexer.queue.startDelay=PT5S

druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

Hi all,

Further more i have attached the full log of completed tasks after being on waiting stage. Please check .

(Attachment Druidingestion.log is missing)

Hi Dhahan,

Sorry to hear that you’re having a problem with task time. If you look at Task reference and scroll down a bit, you’ll find “A description of the fields.” Perhaps this will provide some guidance.

If you have more specifics, please feel free to post them here.

Best,

Mark