[druid-user] avro parsing error details

I'm writing a proof of concept for time-series data ingestion with Kafka, Flink, and Druid. I'm trying to use Avro as my serialization format, but the parsing fails, and I only get this vague exception:

2022-02-12T02:38:23,923 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
org.apache.druid.java.util.common.parsers.ParseException: Fail to decode avro message!
        at org.apache.druid.data.input.avro.InlineSchemaAvroBytesDecoder.parse(InlineSchemaAvroBytesDecoder.java:99) ~[?:?]
        at org.apache.druid.data.input.avro.AvroStreamReader.intermediateRowIterator(AvroStreamReader.java:69) ~[?:?]

I'd like have more exception output so that I can understand more specifically why the parsing failed.

Is there a means to configure that?

Also, which versions of Avro does druid support? I cannot find that anywhere in the documentation.
I am encoding the data with Avro 1.11.0. Would that cause issues?

Also, when I encode avro into ny kafka producer record, I am merely serializing it as a byte array like this:

record = new ProducerRecord<>(
      kafkaTopic,
      null,
      myAvroClass.toByteBuffer.getEventTime().toEpochMilli(),
      null,
      myAvroClass.toByteBuffer.toByteBuffer().array()
);

would that cause problems?

Hi Marco,

I’m looking into your questions, and I’ve come across something which might be worth checking right out of the gate: is there any mismatch between schema and data?

I’ll keep researching your other questions.

Best,

Mark

Hi Marco,

Just ran into this blog post which I think is very relevant:

Let us know how it goes.