Can Druid Ingest Avro Array of Records?

I tried to ingest a file in Avro format. The data are like:
[{
“reqType”: “sp”,
“url”: “https://abcxyz.com/”,
“client”: “customer1”,
“protocol”: “HTTP/1.1”,
“timestamp”: 1624365420,
“httpResponseCode”: 200
}]
[{
“reqType”: “sp”,
“url”: “https://abcxyz.com/”,
“client”: “customer2”,
“protocol”: “HTTP/1.1”,
“timestamp”: 1624365422,
“httpResponseCode”: 200
}]

Each array only has one record.

The Avro schema likes:

{
“name”: “SomeArray”,
“type”: “array”,
“items”: {
“namespace”: “nothing”,
“name”: “ARecord”,
“type”: “record”,
“fields”: [
{
“name”: “reqType”,
“default”: null,
“type”: [
“null”,
“string”
]
},
{
“name”: “urn”,
“default”: null,
“type”: [
“null”,
{
“type”: “map”,
“values”: “string”
}
]
},
{
“name”: “client”,
“default”: null,
“type”: [
“null”,
“string”
]
},
{
“name”: “protocol”,
“default”: null,
“type”: [
“null”,
“string”
]
},
{
“name”: “timestamp”,
“default”: null,
“type”: [
“null”,
“long”
]
},
{
“name”: “httpResponseCode”,
“default”: null,
“type”: [
“null”,
“int”
]
}
]
}
}

I had provided the avro schema (avro.schema.input.value.path) in the jobProperties section.

When I tried to ingest the avro file with avro_hadoop, I got the following error: java.lang.ClassCastException: org.apache.avro.generic.GenericData$Array cannot be cast to org.apache.avro.generic.GenericRecord

That means the avro record reader does not handle the Data Array, but tied to cast to a Record type.

Is it possible to ingest data like above with the current Avro parser?

Any suggestions and/or recommendations are highly appreciated.

Thanks,
Gary

Relates to Apache Druid <0.16.0>

Not sure, maybe it needs to be flattened? The flattenspec is described here. Otherwise, docs say most types are supported, but not “Avro logical types” - not sure what that means, either.

Thanks for the response.

I did have a flattenSpec like:
“flattenSpec”: {
“useFieldDiscovery”: false,
“fields”: [
{
“name”: “req_type”,
“type”: “path”,
“expr”: “$[0].reqType”
},
{
“name”: “urn”,
“type”: “path”,
“expr”: “$[0].urn”
},

The error seems to indicate the AvroRecordReader expects GenericRecord not GenericData$Array. But the data is an array with a single Avro record.

Could you provide a sample AVRO file?

II’ve been trying to recreate a sample using Nifi, and it seems to me that this is not even legal. You would need to have a named type at the outermost scope and array is not a named type.