Looking though some of past questions in the forum concerning the real time nodes I found several answers suggesting that we should be using the Kafka Indexing service instead and that is why no spec file is provided in the 0.10+ releases. Can I get a confirmation on this and an update as to any other druid components that should be preferred over other existing druid components? Could this information also be provided in the documentation on Druid.io?
Realtime ingestion in Druid has gone through a few iterations (realtime nodes -> tranquility -> kafka indexing service with some detours along the way). Right now (Druid 0.10 timeframe) we are in a transitionary period between recommending tranquility and recommending the kafka indexing service. I would say that if the kafka indexing service meets your needs and you are ok with a “rapidly evolving” ingestion scheme, you should go with that. It is where most of the effort is going now, I think it’s the best approach we have taken so far, and I don’t see any reason on the horizon to want to replace it with something else. It has nice operational properties compared to the other two.
If you prize stability above all else then I would go with tranquility at this point in time. Although I expect this advice will change before too long, to just going with the kafka indexing service always.
Definitely agree the docs on druid.io could explain this better. I blame the transitionary period I mentioned earlier
Thanks Gian, This was very helpful!