[druid-user] Data load from Kafka Avro topic to Druid fails

Hi Sushko,

I am actually working with Imply Druid and one of the architect said below when I was working Tranquility along.

“Unless you have a compelling reason to use Tranquility, you should move to the new Kafka Indexing Service. It guarantees exactly once ingestion. There’s a short tutorial here.”

Thanks,

-Madhu

Hi Madhu,
thanks for the reply.

I’ve tried Kafka Indexing Service with following config:

{

“type”: “kafka”,

“dataSchema”: {

“dataSource”: “teststream”,

“parser”: {

“type”: “avro_stream”,

“avroBytesDecoder”: {

“type”: “schema_inline”,

“schema”: {

“namespace”: “io.druid.data”,

“name”: “User”,

“type”: “record”,

“fields”: [

{

“name”: “f1”,

“type”: “string”

},

{

“name”: “timestamp”,

“type”: “long”

},

{

“name”: “f2”,

“type”: “int”

}

]

}

},

“parseSpec”: {

“format”: “timeAndDims”,

“timestampSpec”: {

“column”: “timestamp”,

“format”: “auto”

},

“dimensionsSpec”: {

“dimensions”: [

“f1”,

“f2”

]

}

}

},

“metricsSpec”: ,

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “HOUR”,

“queryGranularity”: “NONE”

}

},

“tuningConfig”: {

“type”: “kafka”,

“maxRowsPerSegment”: 5000000

},

“ioConfig”: {

“topic”: “avroTest0”,

“consumerProperties”: {

“bootstrap.servers”: “http://18.222.199.72:9092

},

“taskCount”: 1,

“replicas”: 1,

“taskDuration”: “PT1M”

}

}

in the Druid Overlord Console I see the same exception in failed tasks:

2019-05-20T17:30:56,540 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
java.lang.UnsupportedOperationException: not supported
	at org.apache.druid.data.input.impl.TimeAndDimsParseSpec$1.parseToMap(TimeAndDimsParseSpec.java:53) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:155) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.data.input.impl.StringInputRowParser.buildStringKeyMap(StringInputRowParser.java:119) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.data.input.impl.StringInputRowParser.parseBatch(StringInputRowParser.java:80) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:50) ~[druid-processing-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:31) ~[druid-processing-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:562) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:166) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]