Kafka Indexing Service doesn't work ("Triggering JVM shutdown.")

Hi,

I’m trying to use Kafka Indexing Service and it doesn’t work.

Environment

My setup (config files and logs you can find here):

All druid nodes started at my local machine in separated docker containers. Containers (version):

  • zookeeper (3.4.6)
  • kafka (0.10.0.1)
  • mysql (5.7.14)
  • druid-coordinator (0.9.1.1)
  • druid-historical (0.9.1.1)
  • druid-broker (0.9.1.1)
  • druid-overlord (0.9.1.1)
    druid-overlord configured to work locally. I have kafka topic with name druid_push_deliver. I double-checked that there is some data in topic. My supervisor config is in file push_deliver.json. I also double-checked that container with druid-overlord can fetch messages from kafka (runned container in interactive mode, downloaded kafka binaries, and successfully run ./kafka-console-consumer)

Problem

Overlord creates task approximately every minute. Task log attach as task_log.txt file. I see that task starts thread task-runner-0 and parse kafka config. I dont’t see any of KafkaSupervisor class mentions in log. After ~1 minute of running task exited with FAILED status, last record in log:

2016-08-24T11:14:00,368 INFO [parent-monitor-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.

I found this info message here.

So I suppose that actually kafka-indexer-task can’t connect to kafka (or can’t read messages), and some of Peon supervisors simply kills task. Also I can’t see kafka-indexing-service in kafka consumers in zookeeper.

Can you help me with this situation? Maybe I can configure overlord to show more detailed logs about kafka-indexing-service extension running?

Thanks in advance and have a nice day!

Hey Aleksandr,

The overlord needs to be able to talk to the peon processes which listen on dynamically opened ports. The starting port is set by druid.indexer.runner.startPort which defaults to 8100 and increments from there. If the overlord can’t contact the peon it will kill the process which is what I suspect is happening.

If you’re still having issues, could you post the logs from the overlord process (which are separate from the task logs) ?

Hi David,

Thanks for fast response.

Overlord and Peon are running in same container. Does Overlord using druid.host property to access Peon? If yes, problem is here – I need additionally open ports when running docker. I’ll check it tomorrow.

Yes it’ll try reaching the peon through the hostname set in druid.host.

Sounds good - let me know how things go.

Hi David,

You was right, problem was in port bindings. I opened ports 8088-8090 (Peon start port is 8088) and now it’s going ok. Thank you!