We are using spark-tranquility to ingest data, and our batch data already reaches >100TB so we decided to use router. For the batch data source, it work really fine and even made huge performance gain when querying recent data. However, for our realtime data source, which has a separate tier from batch, and a dedicated broker as well, we cannot query data still in middlemanager tasks. How are we suppose to setup the broker so it can also watch tiers residing in middleManagers?
Thanks in advance!