kafka ingestion, using messageTime rejection policy

Hi everyone,

I’m using druid 0.9.0 with kafka-eight

Can i ingest data via kafka using messageTime policy without using realtime node, just using kafka as mentioned in related doc.

Best regards,


related doc: http://druid.io/docs/latest/tutorials/tutorial-kafka.html

Hi Yunus,

The tutorial you referenced is an ingestion mechanism based on Tranquility which doesn’t have the concept of messageTime, as Tranquility is designed for a stream of current events. messageTime is not generally used in production systems, and the highly recommended way of handling past data is to set up a batch ingestion pipeline.

For reference, it is possible to modify Tranquility’s understanding of “current” as described here: https://groups.google.com/forum/#!searchin/druid-user/messageTime/druid-user/qm0EqwpUIyo/qwsxNk86AAAJ - but Tranquility-Kafka doesn’t currently expose this functionality.