Setting up a local kafka/druid system for testing


I set up a small kafka/druid system using docker-compose on my workstation.

I pull data into druid using druid-kafka-indexing-service, but it seems that the information about that date propagates very slowly between the different druid parts.

On the Coordinator Console I can see the workers feeling well, but it takes quite a few minutes until the datasource is visible in the “old-console” and more like hours until it is visible in the main druid console and the coordinator API.

Any idea how I could debug this?

I have the feeling that it might be because of limited ressources on my workstation (32G RAM, 12 CPUs, no swap), but according to htop CPU and RAM don’t seem to be the problem.

Are there any good configuration values, I should try, which could lead to a more “fluent” behavior of the system?


I have things setup on vagrant, and the VM is only 8 gig with 8 cores. I have picked conf from conf-quickstart and it seems to be running fine fairly.