Roll Ups and Raw Data?

I’m not sure If I’m understanding the ingestion/roll up fully,
Note: I am ingesting data via Kafka-Indexer.

My question is:

Can I do both at the same time with the same spec?

i.e. I want to ingest the raw data AND store it, but simultaneously I would like to also do roll-ups (ideally to a different Data Source).

For example:

I would like to ingest data to ‘raw_events’, while at the same time have the data ‘aggregated/rolled-up’ to say an event_summary datasource.

From what I have been led to believe, to do this I would need to define 2 datasource ingestion specs, meaning I need to process the same data twice from kafka. If there a way to do it ‘once’?

Thanks

Dan

Hey Dan,

Setting up two ingestions into two different datasources is the best way to do this. Right now, the same ingestion job splitting a single input topic into multiple output datasources isn’t supported. (But two jobs can read the same topic.)