How to set timestampSpec and dimensionsSpec for multiple_schemas_inline

Hi,
I am using multiple_schemas_inline , while reading different events from the same kafka topic to store under single datasource in Druid.

I am able to read data from 2 different avro’s using Inline Schema Based Avro Bytes Decoder one at a time and each avro has their own dimensionsSpecs: one having “dimensionsSpec”: { “dimensions”: [A, B, C ] } and other having “dimensionsSpec”: { “dimensions”: [D, E, F ] }. And both having same timestampSpec as : “timestampSpec”: {“column”: “timestamp”,“format”: “auto”} .

When i try to modify avroBytesDecoder for “type”: “multiple_schemas_inline”, i am able to merge both schema’s as :

“schemas”: {“1” : {schema1}, “2”: {Schema2}}

I am having difficulty to merge both avro’s dimensionsSpec’s to one. Can you please help me out in understanding , how to write dimensionsSpec’s in case of multiple_schemas_inline.

Thanks & Regards,

Jagadesh.A

The assumption is that both schemas are really the “same” kind of object with some kind of evolution happening. So that’s why there’s only one dimensionsSpec. You might be able to get away with specifying a superset of dimensions.

Gian,
I tried with superset of dimensions. Some how it’s not able to read the 2 different events data. Any alternative approach, that could help me out.

Thanks & Regards,

Jagadesh

Hmm, perhaps check the task counters to see if events are showing up as unparseable. Are you including the extra header for multiple_schemas_inline described on http://druid.io/docs/latest/development/extensions-core/avro.html? If not, then Druid won’t know which schema to use.

Gian,
I tried to add extra header for multiple_schemas_inline as described on http://druid.io/docs/latest/development/extensions-core/avro.html. Could you please verify attached superviser-spec.json, if i am missing something.

Thanks & Regards,

Jagadesh

supervisor-spec.json (2.71 KB)

Hi Gian,

Could you please help me in verifying attached superviser-spec.json for multiple_schemas_inline and let me know if something something or it’s a bug?.

Thanks & Regards,

Jagadesh.A