All batch ingestions tasks picked up by only one host

Hi,

I am new to Druid and was doing a PoC for the same. I generated some dataset using

for i in $(seq 1 100) ; do curl -O http://static.druid.io/data/benchmarks/tpch/100/lineitem.tbl.$i.gz ; done

I uploaded these to s3 and started 3 ingestions tasks (three of the above files). It looks like every ingestion is happening sequentially (from the attached screenshot). Here is the middle manage and overlord configurations:

overlod,

druid.host={config.hostname}

druid.port=8090

druid.service=druid/prod/overlord

Remote workers (middleManager + peons)

druid.indexer.runner.type=remote

Allows overlord restart or other overlord take leadership

druid.indexer.storage.type=metadata

middle-manager

druid.host={config.hostname}

druid.port=8084

druid.service=druid/prod/middlemanager

Index task log dir

druid.indexer.logs.directory={config.task-log-dir}

Resources for peons

#druid.indexer.runner.javaOpts=-server -Xmx25g -Xms1g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

druid.indexer.task.baseTaskDir={config.base-task-dir}

druid.indexer.task.restoreTasksOnRestart=true

Peon properties

druid.indexer.fork.property.druid.monitoring.monitors=[“com.metamx.metrics.JvmMonitor”]

druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912

druid.indexer.fork.property.druid.processing.numThreads=2

druid.indexer.task.defaultRowFlushBoundary = 50000

druid.indexer.fork.property.druid.segmentCache.locations=[{“path”: “{config.segment-cache-dir}”, “maxSize”: 300000000000}]

druid.indexer.fork.property.druid.server.http.numThreads=50

druid.worker.capacity=10

is this expected behavior or Am I missing some configuration? Any help would be appreciated.

-Theja

These are proceeding in parallel (they’re all “running”) but they’re in parallel on a single host. Which is fine, because a host can run multiple tasks. If you prefer to spread the tasks out, you can check out “worker select strategies” here: http://druid.io/docs/latest/configuration/indexing-service.html

Hi Gian,

Thank you! That makes sense.

Cheers,

Theja