I am trying to leverage tranquility with kafka hose to ingest data into datasources in real time,
I have this use case where there are n datasources(can increase on a regular basis), apart from a common set of dimensions, these datasources have their own unique set of dimensions respectively. This unique set may differ completely for every data source, some new dimensions may be added to this set over time, thereby I am fine with defining ‘’ when ingesting data rather than specifying a data type.
So, the issue is,
when i ingest data with ‘’ as dimensions in dimensionSpec, I am unable to specify datatypes for the common set
when i ingest data with ‘dimensions specified’ in dimensionSpec, I am unable to specify datatypes that add over time
Can you please suggest how would I proceed to solve this?