I am trying to ingest the log messages coming via MQTT queue to Druid using Tranquility with Spark Streaming.
The flow is like this:
Spark Streaming which receives MQTT log events => Convert to Tranquility’s Beam RDD and send to Druid
The tuning config used in the Tranquility’s Beam
ClusteredBeamTuning.create( Granularity.FIVE_MINUTE, new Period("PT1M"), new Period("PT4M"), 1, 1)
The realtime ingestion task is created and started as soon as the events are pushed to MQTT. I can see the task running in the coordinator console. But the corresponding data source will not be created until the task is completed. And this task takes approximately 9 minutes (5 + 4 minutes) to complete only after which, I can query for the data.
According to the documentation I read so far, realtime data will also be available for querying. Can somebody tell me why am I not able to query the realtime data here?
Please do reply for any extra information needed to narrow down the issue.