Hi Sushko,
I am actually working with Imply Druid and one of the architect said below when I was working Tranquility along.
“Unless you have a compelling reason to use Tranquility, you should move to the new Kafka Indexing Service. It guarantees exactly once ingestion. There’s a short tutorial here.”
Thanks,
-Madhu
Hi Madhu,
thanks for the reply.
I’ve tried Kafka Indexing Service with following config:
{
“type”: “kafka”,
“dataSchema”: {
“dataSource”: “teststream”,
“parser”: {
“type”: “avro_stream”,
“avroBytesDecoder”: {
“type”: “schema_inline”,
“schema”: {
“namespace”: “io.druid.data”,
“name”: “User”,
“type”: “record”,
“fields”: [
{
“name”: “f1”,
“type”: “string”
},
{
“name”: “timestamp”,
“type”: “long”
},
{
“name”: “f2”,
“type”: “int”
}
]
}
},
“parseSpec”: {
“format”: “timeAndDims”,
“timestampSpec”: {
“column”: “timestamp”,
“format”: “auto”
},
“dimensionsSpec”: {
“dimensions”: [
“f1”,
“f2”
]
}
}
},
“metricsSpec”: ,
“granularitySpec”: {
“type”: “uniform”,
“segmentGranularity”: “HOUR”,
“queryGranularity”: “NONE”
}
},
“tuningConfig”: {
“type”: “kafka”,
“maxRowsPerSegment”: 5000000
},
“ioConfig”: {
“topic”: “avroTest0”,
“consumerProperties”: {
“bootstrap.servers”: “http://18.222.199.72:9092”
},
“taskCount”: 1,
“replicas”: 1,
“taskDuration”: “PT1M”
}
}
in the Druid Overlord Console I see the same exception in failed tasks:
2019-05-20T17:30:56,540 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
java.lang.UnsupportedOperationException: not supported
at org.apache.druid.data.input.impl.TimeAndDimsParseSpec$1.parseToMap(TimeAndDimsParseSpec.java:53) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:155) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.data.input.impl.StringInputRowParser.buildStringKeyMap(StringInputRowParser.java:119) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.data.input.impl.StringInputRowParser.parseBatch(StringInputRowParser.java:80) ~[druid-core-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:50) ~[druid-processing-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:31) ~[druid-processing-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:562) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:166) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.14.1-incubating.jar:0.14.1-incubating]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]