Parse Spec for ingesting smile encoded messages

Hey folks,

We have a realtime node reading directly from a kafka queue. When the messages are in json format, it works fine but if I switch to another topic in which smile encoded messages are written, it doesn’t work and I can see the message " Unparsable Event " received.

The parse spec is :

“parser” : {

“type” : “string”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column” : “ts”,

“format” : “auto”

},

“dimensionsSpec” : {

“dimensions”: [“page”,“svcName”],

“dimensionExclusions” : ,

“spatialDimensions” :

}

}

}

Is there any different type of parses or different format spec I need to specify ?

Can someone please point me to any documentation I might be missing ?

Any pointers will be of great help.

Thanks,

Tejas

Hi Tejas, you may have to write a custom parser for smile encoded messages.

A good starting point is to look at how Druid actually reads the message from Kafka:

https://github.com/druid-io/druid/blob/master/extensions/kafka-eight/src/main/java/io/druid/firehose/kafka/KafkaEightFirehoseFactory.java#L118

The StringInputRowParser and ProtoBufInputRowParser provide examples of how to parse the message.