Thrift data ingestion using Kafka indexing services

hi all ,

I am trying to ingest the thrift formatted Kafka topic , In to druid using Kafka indexing services .

I did not find any proper documentation for that , so I followed whatever is available in http://druid.io/docs/latest/development/extensions-contrib/thrift.html

here is my ingestion spec :

{

"type": “kafka”,

"dataSchema": {

"dataSource": “datasourcename”,

"parser": {

"type": “thrift”,

"thriftClass": “org.apache.druid.data.input.thrift.Book”,

"parseSpec": {

"format": “json”,

"timestampSpec": {

"column": “isoFormatDate”,

"format": "auto"

},

"dimensionsSpec": {

"dimensions": [

"col1" ,

"col2",

"col3"

]

}

}

},

"metricsSpec" : [

],

"granularitySpec": {

"type": “uniform”,

"segmentGranularity": “HOUR”,

"queryGranularity": "MINUTE"

}

},

"ioConfig": {

"topic": “topic name”,

"consumerProperties": {

"bootstrap.servers": "hostname and port"

},

"taskDuration": “PT20M”,

"lateMessageRejectionPeriod": "PT1H"

}

}

When I submit the above ingestion spec , all my events are getting under "unparsable "

the status of the indextask is :

{“ingestionStatsAndErrors”:{“taskId”:“index_kafka_datasourcename_b01a2a13eca1d05_mooloald”,“payload”:{“ingestionState”:“COMPLETED”,“unparseableEvents”:{},“rowStats”:{“buildSegments”:{“processed”:0,“processedWithError”:0,“thrownAway”:0,“unparseable”:4521632}},“errorMsg”:null},“type”:“ingestionStatsAndErrors”}}

Does anyone know what I am doing wrong here ??

Thanks,

Anoosha

Hi All,

any takes on this ?

Thanks,

Anoosha

Can anyone who is using thrift format data ingestion through Kafka to druid , connect with me please ?

Thanks,

Anoosha

Bumping this

Thanks,

Anoosha

See if this example helps https://github.com/apache/incubator-druid/blob/master/extensions-contrib/thrift-extensions/example/books.json

Thanks,

Sashi

Hi Anoosha,

We have also recently using druid, you can probably try setting logparexceptions true in MM or Overload process.

Thanks

Sumanshu

Hi. Sashi.,

I hva e a doubt in the example you sent.

What is supposed to be as the thrift class value in our supervisor spec?

““org.apache.druid.data.input.thrift.Book”,”,

Does it mean I Should be writing the separate class, with the columns I am ingesting in one of the. Java files inside thrift extension jar???

Thanks, Anoosha

As I understand it, you should be creating a jar of the Thrift generated classes. Provide the jar path and the fully qualified thrift class name in the spec.

Thanks,

Sashi