Data availability in streaming ingestion

Hi guys,

I’m a little confused on when data becomes available for querying when ingested in real-time. The overview documentation claims that the data is available immediately. However I can’t see new datasource after starting Tranquility server and sending few messages. I sent them with fresh timestamps and I see in Tranquility logs that they are received. So do I have to wait until the segment is handed off to Historical node? If so, then it is not really real-time ingestion and the documentation shouldn’t claim immediate data availability.

Here’s my setup:
druid-0.9.2

tranquility-distribution-0.8.0

“segmentGranularity” : “day”

“windowPeriod” : “PT1H”

Do I need to wait for 25 hours to see my data?

Thank you.

Artem,

If the data is pre loaded it is already there as well as if caching is enabled. Then, data is available immediately. If you are running new queries the query will take time to crunch; if it is not cached. Data will be loaded from deep storage according to metadata storage segments table.

Thanks for reply. What exactly do you mean by preloaded? How can real time data be preloaded? We can’t see future yet :slight_smile: