Conditional indexing in different DataSources in Druid

Apache Druid Q&A Google Groups
#1

Hello,
I have heterogeneous json files which have json objects with different schemas present together. Now i want to ingest these json objects in different data sources according to some parameter (maybe some value for a particular key present in all the json objects). So that each datasource has only a particular class of json objects with same schema.

For example. there is a file to be ingested.

{ “type” : “a” , “property1” : “xyz” }

{ “type” : “b” , “property2”: “abc” , “property3” : “def” }

Now i want all the json objects with type : a to be ingested in a single datasource A and all the json objects with type : b to be ingested in datasource B. such that both A and B have different schemas .

Is there any possible way to do this or any other possible closest methods through which i can somehow achieve my intended result.

Thanks for your help,

Shubham

#2

Hey Shubham,

Transform specs can do this - check out the “filter” option: http://druid.io/docs/latest/ingestion/transform-spec.html

#3

Hey Gian,
Thanks a lot for your help.

Shubham.