Roll Ups and Raw Data?

I’m not sure If I’m understanding the ingestion/roll up fully,
Note: I am ingesting data via Kafka-Indexer.

My question is:

Can I do both at the same time with the same spec?

i.e. I want to ingest the raw data AND store it, but simultaneously I would like to also do roll-ups (ideally to a different Data Source).

For example:

I would like to ingest data to ‘raw_events’, while at the same time have the data ‘aggregated/rolled-up’ to say an event_summary datasource.

From what I have been led to believe, to do this I would need to define 2 datasource ingestion specs, meaning I need to process the same data twice from kafka. If there a way to do it ‘once’?



Hey Dan,

Setting up two ingestions into two different datasources is the best way to do this. Right now, the same ingestion job splitting a single input topic into multiple output datasources isn’t supported. (But two jobs can read the same topic.)