Ingest data to multiple datasource using values defined in dimensions ( data points)


I want to ingest data to datasource based on datapoint attribute value.

Data Point pushing to kafka : {“key”:“a1”,“value”:“1”,datasources:[“raw_data”,“rollback_1hr”]}

key and datasources are dimensions

i already created raw_data and rollback_1hr supervisor task with type kafka and given rollback as false for raw_data and rollback as true for rollback_1hr. Using same topic for both supervisor.

Now i want to ingest above datapoint to both datasources, Data Sources values might change based on some check done in backend .

Basically needs to read datasources from data point ( dimension ) and ingest to those data sources.

is there any way to ingest data to data sources that defined in datapoint ?

Hi @Navin,

Are you thinking of applying a filter during ingestion?



Hey Navin,

Another idea would be to process the incoming stream by feeding the key/values to separate topics, one per data source target. Then have an ingestion job for each topic/datasource.

The latter option is the right way to do it, if that is not possible, then the former is the way to do it if you had to.