Push model with tranquility & index service

I am implementing pushed based indexing model with druid index service.

I want to use index service to ingest data from both real time and batch sources. This will be done using tranquility. For real time data, I will be listening to rabbitmq and push data using tranquility finagle inteface. For batch operation I will be reading data from cassandra and push it using tranquility finagle interface.

My question is how to differentiate between realtime and batch operations ? Is there a way to configure DruidBeam to do, or it does not matter ?

Hi Ravish,

Druid realtime ingestion in general, and tranquility in particular, doesn’t distinguish between streams and batches. Batches you send with tranquility are broken up and ingested the same way as streams.

One thing to keep in mind here is that windowPeriod config can cause you some issues if you’re trying to ingest older data. If you are doing that, then you should use one of the true batch ingestion methods (“index” task for moderate sized file uploads, hadoop indexing for big datasets). They have no windowPeriod limit.

Hope this helps.

Thanks Gian, this helps !!