Exception while ingesting protobuf data

Hi,

I am using apache-druid-0.14.2-incubating version. We have lots of log files written in protobuf format. We want to ingest the same into druid by using druid-protobuf-extensions. I have creaated the below ingestion file :

{
“type”: “index”,
“spec”: {
“dataSchema”: {
“dataSource”: “procProto”,
“parser”: {
“type”: “protobuf”,
“descriptor”: “/root/himanshu/processor_log.proto”,
“parseSpec”: {
“format”: “json”,
“dimensionsSpec”: {
“dimensions”: [
“agid”
]
},
“timestampSpec”: {
“column”: “timestamp”,
“format”: “posix”
}
}
},
“metricsSpec”: [
{
“name”: “uniq_user”,
“type”: “thetaSketch”,
“fieldName”: “userId”
}
],
“granularitySpec”: {
“type”: “uniform”,
“segmentGranularity”: “day”,
“queryGranularity”: “minute”,
“rollup”: true
}
},
“ioConfig”: {
“type”: “index”,
“firehose”: {
“type”: “local”,
“baseDir”: “/root/himanshu/”,
“filter”: “procData”
},
“appendToExisting”: false
},
“tuningConfig”: {
“type”: “index”,
“maxRowsPerSegment”: 5000000,
“maxRowsInMemory”: 25000,
“forceExtendableShardSpecs”: true
}
}
}

I tried ingesting using the above spec file and I got the below exception, not sure why it is coming.

2019-07-03T10:24:17,541 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in DETERMINE_PARTITIONS.
java.lang.ClassCastException: org.apache.druid.segment.transform.TransformingInputRowParser cannot be cast to org.apache.druid.data.input.impl.StringInputRowParser

Any help is appreciated.

One possibility could be that the protobuf extension is not loaded, Can you check if you have druid-protobuf-extensions loaded.

And couple of things that stands out when reading your spec with the instructions here https://druid.apache.org/docs/latest/development/extensions-core/protobuf.html are

  1. The doc says to give .desc file in “descriptor”: “file:///tmp/metrics.desc”, I see that you are giving the .proto file directly. And you can generate the .desc file using protc protoc -o /tmp/metrics.desc metrics.proto not sure if this will make any difference

  2. And also the document says “Protobuf descriptor file name in the classpath or URL.” the descriptor file should be in classpath or URL. So either give file:/// or put the descriptor in the druid classpath

Hi Muthu,

I have the druid-protobuf-extensions loaded.

I have already tried with .desc file which I generated using the protoc command, then I tried with proto file itself; both of them did not work.

How do I put the descriptor file in the classpath? Earlier when I tried to put the descriptor, I did it like this ’ “descriptor”: “/root/himanshu/proc.desc”,’