Hi,
I’m trying to launch my first Kafka indexing service with imply 2.9.6.
When I started the druid it failed to start because of caffeine-cache wasn’t installed so remove it from the comm props.
Moreover, the router service failed to start so I comment it out, I don’t need the router for now.
The Druid launched successfully
I sent indexing service task with bellow configuration, and most of my tasks failed
attached log of one of my failed tasks.
Do you have any idea why it failed?
thx
Indexing configuration
{
``
“type”: “kafka”,
“dataSchema”: {
“dataSource”: “enriched-test”,
“parser”: {
“type”: “string”,
“parseSpec”: {
“format”: “json”,
“timestampSpec”: {
“column”: “timestamp”,
“format”: “auto”
},
“dimensionsSpec”: {
“dimensions”: [
…
],
“format”: “json”
}
}
},
“metricsSpec”: [
{
“type” : “count”,
“name” : “totalEvents”
}, {
“type” : “hyperUnique”,
“name” : “uniqueUsers”,
“fieldName” : “userHash”
}, {
“type” : “longMax”,
“name” : “lastOccurrence”,
“fieldName” : “__time”
}, {
“type” : “hyperUnique”,
“name” : “uniqueEvents”,
“fieldName” : “id”
}
],
“granularitySpec”: {
“segmentGranularity” : “day”,
“type” : “uniform”,
“queryGranularity” : “hour”
}
},
“tuningConfig”: {
“maxRowsInMemory” : “100000”,
“type” : “kafka”,
“windowPeriod” : “PT50M”,
“intermediatePersistPeriod” : “PT10M”,
“rejectionPolicy” : {
“type”: “messageTime”
}
},
“ioConfig”: {
“topic”: “enriched-all”,
“consumerProperties”: {
“bootstrap.servers”: “x.x.x.x:9092”
},
“taskCount”: 3,
“replicas”: 1,
“taskDuration”: “PT1H”,
“useEarliestOffset”: true
}
}
Failed task
egment.StringDimensionMergerV9 - Completed dim[utmSource] inverted with cardinality[1,653] in 245 millis.
2019-05-24T13:34:29,865 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [58,540] bytes of dictionary merging direct buffers
2019-05-24T13:34:30,268 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionCount] inverted with cardinality[4,866] in 399 millis.
2019-05-24T13:34:30,268 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [101,832] bytes of dictionary merging direct buffers
2019-05-24T13:34:30,458 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[eventType] inverted with cardinality[3] in 177 millis.
2019-05-24T13:34:30,458 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [648] bytes of dictionary merging direct buffers
2019-05-24T13:34:31,058 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[initialReferrer] inverted with cardinality[50,391] in 600 millis.
2019-05-24T13:34:31,058 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [336,268] bytes of dictionary merging direct buffers
2019-05-24T13:34:31,430 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[initialReferrerDomain] inverted with cardinality[6,735] in 347 millis.
2019-05-24T13:34:31,430 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [104,580] bytes of dictionary merging direct buffers
2019-05-24T13:34:32,086 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userReferrer] inverted with cardinality[54,932] in 646 millis.
2019-05-24T13:34:32,086 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [366,192] bytes of dictionary merging direct buffers
2019-05-24T13:34:32,514 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userReferrerDomain] inverted with cardinality[6,980] in 401 millis.
2019-05-24T13:34:32,514 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [111,820] bytes of dictionary merging direct buffers
2019-05-24T13:34:32,837 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionUtmCampaign] inverted with cardinality[10,514] in 312 millis.
2019-05-24T13:34:32,837 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [176,864] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,086 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionUtmMedium] inverted with cardinality[817] in 242 millis.
2019-05-24T13:34:33,086 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [35,792] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,352 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionUtmSource] inverted with cardinality[1,561] in 262 millis.
2019-05-24T13:34:33,352 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [57,148] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,684 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userUtmCampaign] inverted with cardinality[11,707] in 326 millis.
2019-05-24T13:34:33,684 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [194,940] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,938 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userUtmMedium] inverted with cardinality[964] in 247 millis.
2019-05-24T13:34:33,938 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [38,088] bytes of dictionary merging direct buffers
2019-05-24T13:34:34,207 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userUtmSource] inverted with cardinality[1,842] in 265 millis.
2019-05-24T13:34:34,208 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [62,656] bytes of dictionary merging direct buffers
2019-05-24T13:34:34,832 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTReferrer] inverted with cardinality[53,028] in 619 millis.
2019-05-24T13:34:34,832 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [342,688] bytes of dictionary merging direct buffers
2019-05-24T13:34:35,204 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTReferrerDomain] inverted with cardinality[6,883] in 346 millis.
2019-05-24T13:34:35,204 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [103,720] bytes of dictionary merging direct buffers
2019-05-24T13:34:35,584 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTUtmCampaign] inverted with cardinality[11,905] in 370 millis.
2019-05-24T13:34:35,584 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [199,864] bytes of dictionary merging direct buffers
2019-05-24T13:34:35,839 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTUtmMedium] inverted with cardinality[972] in 247 millis.
2019-05-24T13:34:35,839 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [38,476] bytes of dictionary merging direct buffers
2019-05-24T13:34:36,111 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTUtmSource] inverted with cardinality[1,871] in 266 millis.
2019-05-24T13:34:36,111 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [63,196] bytes of dictionary merging direct buffers
2019-05-24T13:34:36,116 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMergerV9 - Completed index.drd in 0 millis.
2019-05-24T13:34:36,116 INFO [appenderator_merge_0] org.apache.druid.java.util.common.io.smoosh.FileSmoosher - Created smoosh file [/opt/imply/var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged/00000.smoosh] of size [654104306] bytes.
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.segment.loading.LocalDataSegmentPusher - Copying segment[enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z] to local filesystem at location[var/druid/segments/enriched-test/2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z/2019-05-24T12:09:09.879Z/0/f11acda0-4746-42f6-be61-62a1e2f67d27]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.segment.loading.LocalDataSegmentPusher - Creating intermediate directory[var/druid/segments/intermediate_pushes/dda1c20a-2afc-480c-b2c8-dc0fc4325d18] for segment[enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.segment.loading.LocalDataSegmentPusher - Compressing files from[var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged] to [var/druid/segments/intermediate_pushes/dda1c20a-2afc-480c-b2c8-dc0fc4325d18/index.zip]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.java.util.common.CompressionUtils - Adding file[var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged/factory.json] with size[29]. Total size so far[0]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.java.util.common.CompressionUtils - Adding file[var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged/00000.smoosh] with size[654,104,306]. Total size so far[29]
``