Is Protobuf and Avro Data Format Supported for Ingestion via Kafka Indexing Service

Hi,

I see in following doc that it mentions only about Realtime ingestion and Hadoop Indexor for using Avro Data Format. Do we have Avro Data format support for Kafka Indexing Service as well?

http://druid.io/docs/latest/development/extensions-core/avro.html

And w.r.t Protobuf, Does Druid Support it ? I dont see any doc for the same.

Thanks,

Pravesh Gupta

Hi Pravesh,

I think Avro parser should work for Kafka indexing service as well.

For Protobuf parser, please refer to http://druid.io/docs/latest/ingestion/.

Also, our protobuf support will be improved in 0.10.1. You can find more information at https://github.com/druid-io/druid/blob/0.10.1/docs/content/development/extensions-core/protobuf.md.

You can download 0.10.1-rc1 from http://static.druid.io/artifacts/releases/druid-0.10.1-rc1-bin.tar.gz.

Thanks,

Jihoon

2017년 7월 1일 (토) 오전 1:46, Pravesh Gupta gupta.praveshg@gmail.com님이 작성:

Thanks for the Confirmation Jihoon.

Hi Jihoon,

Quick Question ,

I am using imply-2.2.3 to use Druid. And I am using Proto 3 Parser for using protobuf data formats.

What version of Druid it has. Is Proto 3 supported in above imply-2.2.3 because I do not happen to ingest data into Druid using this configuration.

Hi Pravesh,

imply-2.2.3 is based on druid 0.10.0, and proto3 should work.
I left a comment on https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/imply-user-group/0c8WtyU5FX4/OrogAtkoBQAJ.

Jihoon

2017년 7월 12일 (수) 오후 4:26, Pravesh Gupta gupta.praveshg@gmail.com님이 작성:

Hi Jihoon,

Looks like the Protobuf 3 Support was added only in 0.10.1 .Hence with the above imply-2.2.3 , Protobuf would not work .And the protobuf extension is also not loaded by druid.

https://github.com/druid-io/druid/pull/4039 — > This says the same.

Thanks,

Pravesh Gupta

I’m not sure what the author of that patch wanted to say about protobuf version migration. The protobuf version was actually migrated from 2 to 3 in https://github.com/druid-io/druid/pull/3682. You can see the version is already 3.1.0 in 0.10.0 release (https://github.com/druid-io/druid/blob/druid-0.10.0/pom.xml#L529).

Before 0.10.1, the protobuf extension didn’t exist, but you can still use the protobuf parser.

2017년 7월 12일 (수) 오후 6:53, Pravesh Gupta gupta.praveshg@gmail.com님이 작성:

Well I can see in the Release Notes of Druid for 0.10.1 also , they have mentioned about the Protobuf 3.0 Support . May be it was broken in 0.10.0 .

See for yourself : https://github.com/druid-io/druid/issues/4384

Btw, Any clue when the 0.10.1 release is going to come out.

Thanks,

Pravesh Gupta