I’m trying to use Kafka Indexing Service and it doesn’t work.
My setup (config files and logs you can find here):
All druid nodes started at my local machine in separated docker containers. Containers (version):
- zookeeper (3.4.6)
- kafka (0.10.0.1)
- mysql (5.7.14)
- druid-coordinator (0.9.1.1)
- druid-historical (0.9.1.1)
- druid-broker (0.9.1.1)
- druid-overlord (0.9.1.1)
druid-overlord configured to work locally. I have kafka topic with name druid_push_deliver. I double-checked that there is some data in topic. My supervisor config is in file push_deliver.json. I also double-checked that container with druid-overlord can fetch messages from kafka (runned container in interactive mode, downloaded kafka binaries, and successfully run ./kafka-console-consumer)
Overlord creates task approximately every minute. Task log attach as task_log.txt file. I see that task starts thread task-runner-0 and parse kafka config. I dont’t see any of KafkaSupervisor class mentions in log. After ~1 minute of running task exited with FAILED status, last record in log:
2016-08-24T11:14:00,368 INFO [parent-monitor-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.
I found this info message here.
So I suppose that actually kafka-indexer-task can’t connect to kafka (or can’t read messages), and some of Peon supervisors simply kills task. Also I can’t see kafka-indexing-service in kafka consumers in zookeeper.
Can you help me with this situation? Maybe I can configure overlord to show more detailed logs about kafka-indexing-service extension running?
Thanks in advance and have a nice day!