My First Kafka Indexing service - tasks failure

Hi,
I’m trying to launch my first Kafka indexing service with imply 2.9.6.
When I started the druid it failed to start because of caffeine-cache wasn’t installed so remove it from the comm props.
Moreover, the router service failed to start so I comment it out, I don’t need the router for now.
The Druid launched successfully :slight_smile:

I sent indexing service task with bellow configuration, and most of my tasks failed :frowning:
attached log of one of my failed tasks.
Do you have any idea why it failed?
thx

Indexing configuration
{

``
“type”: “kafka”,

“dataSchema”: {

“dataSource”: “enriched-test”,

“parser”: {

“type”: “string”,

“parseSpec”: {

“format”: “json”,

“timestampSpec”: {

“column”: “timestamp”,

“format”: “auto”

},

“dimensionsSpec”: {

“dimensions”: [

],

“format”: “json”

}

}

},

“metricsSpec”: [

{

“type” : “count”,

“name” : “totalEvents”

}, {

“type” : “hyperUnique”,

“name” : “uniqueUsers”,

“fieldName” : “userHash”

}, {

“type” : “longMax”,

“name” : “lastOccurrence”,

“fieldName” : “__time”

}, {

“type” : “hyperUnique”,

“name” : “uniqueEvents”,

“fieldName” : “id”

}

],

“granularitySpec”: {

“segmentGranularity” : “day”,

“type” : “uniform”,

“queryGranularity” : “hour”

}

},

“tuningConfig”: {

“maxRowsInMemory” : “100000”,

“type” : “kafka”,

“windowPeriod” : “PT50M”,

“intermediatePersistPeriod” : “PT10M”,

“rejectionPolicy” : {

“type”: “messageTime”

}

},

“ioConfig”: {

“topic”: “enriched-all”,

“consumerProperties”: {

“bootstrap.servers”: “x.x.x.x:9092”

},

“taskCount”: 3,

“replicas”: 1,

“taskDuration”: “PT1H”,

“useEarliestOffset”: true

}

}

Failed task

egment.StringDimensionMergerV9 - Completed dim[utmSource] inverted with cardinality[1,653] in 245 millis.
2019-05-24T13:34:29,865 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [58,540] bytes of dictionary merging direct buffers
2019-05-24T13:34:30,268 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionCount] inverted with cardinality[4,866] in 399 millis.
2019-05-24T13:34:30,268 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [101,832] bytes of dictionary merging direct buffers
2019-05-24T13:34:30,458 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[eventType] inverted with cardinality[3] in 177 millis.
2019-05-24T13:34:30,458 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [648] bytes of dictionary merging direct buffers
2019-05-24T13:34:31,058 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[initialReferrer] inverted with cardinality[50,391] in 600 millis.
2019-05-24T13:34:31,058 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [336,268] bytes of dictionary merging direct buffers
2019-05-24T13:34:31,430 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[initialReferrerDomain] inverted with cardinality[6,735] in 347 millis.
2019-05-24T13:34:31,430 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [104,580] bytes of dictionary merging direct buffers
2019-05-24T13:34:32,086 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userReferrer] inverted with cardinality[54,932] in 646 millis.
2019-05-24T13:34:32,086 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [366,192] bytes of dictionary merging direct buffers
2019-05-24T13:34:32,514 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userReferrerDomain] inverted with cardinality[6,980] in 401 millis.
2019-05-24T13:34:32,514 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [111,820] bytes of dictionary merging direct buffers
2019-05-24T13:34:32,837 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionUtmCampaign] inverted with cardinality[10,514] in 312 millis.
2019-05-24T13:34:32,837 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [176,864] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,086 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionUtmMedium] inverted with cardinality[817] in 242 millis.
2019-05-24T13:34:33,086 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [35,792] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,352 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sessionUtmSource] inverted with cardinality[1,561] in 262 millis.
2019-05-24T13:34:33,352 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [57,148] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,684 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userUtmCampaign] inverted with cardinality[11,707] in 326 millis.
2019-05-24T13:34:33,684 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [194,940] bytes of dictionary merging direct buffers
2019-05-24T13:34:33,938 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userUtmMedium] inverted with cardinality[964] in 247 millis.
2019-05-24T13:34:33,938 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [38,088] bytes of dictionary merging direct buffers
2019-05-24T13:34:34,207 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[userUtmSource] inverted with cardinality[1,842] in 265 millis.
2019-05-24T13:34:34,208 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [62,656] bytes of dictionary merging direct buffers
2019-05-24T13:34:34,832 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTReferrer] inverted with cardinality[53,028] in 619 millis.
2019-05-24T13:34:34,832 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [342,688] bytes of dictionary merging direct buffers
2019-05-24T13:34:35,204 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTReferrerDomain] inverted with cardinality[6,883] in 346 millis.
2019-05-24T13:34:35,204 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [103,720] bytes of dictionary merging direct buffers
2019-05-24T13:34:35,584 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTUtmCampaign] inverted with cardinality[11,905] in 370 millis.
2019-05-24T13:34:35,584 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [199,864] bytes of dictionary merging direct buffers
2019-05-24T13:34:35,839 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTUtmMedium] inverted with cardinality[972] in 247 millis.
2019-05-24T13:34:35,839 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [38,476] bytes of dictionary merging direct buffers
2019-05-24T13:34:36,111 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[tokenFTUtmSource] inverted with cardinality[1,871] in 266 millis.
2019-05-24T13:34:36,111 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Freed [63,196] bytes of dictionary merging direct buffers
2019-05-24T13:34:36,116 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMergerV9 - Completed index.drd in 0 millis.
2019-05-24T13:34:36,116 INFO [appenderator_merge_0] org.apache.druid.java.util.common.io.smoosh.FileSmoosher - Created smoosh file [/opt/imply/var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged/00000.smoosh] of size [654104306] bytes.
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.segment.loading.LocalDataSegmentPusher - Copying segment[enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z] to local filesystem at location[var/druid/segments/enriched-test/2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z/2019-05-24T12:09:09.879Z/0/f11acda0-4746-42f6-be61-62a1e2f67d27]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.segment.loading.LocalDataSegmentPusher - Creating intermediate directory[var/druid/segments/intermediate_pushes/dda1c20a-2afc-480c-b2c8-dc0fc4325d18] for segment[enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.segment.loading.LocalDataSegmentPusher - Compressing files from[var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged] to [var/druid/segments/intermediate_pushes/dda1c20a-2afc-480c-b2c8-dc0fc4325d18/index.zip]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.java.util.common.CompressionUtils - Adding file[var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged/factory.json] with size[29].  Total size so far[0]
2019-05-24T13:34:36,211 INFO [appenderator_merge_0] org.apache.druid.java.util.common.CompressionUtils - Adding file[var/druid/task/index_kafka_enriched-test_dc66e3ddcd21495_bebmfiod/work/persist/enriched-test_2019-05-17T00:00:00.000Z_2019-05-18T00:00:00.000Z_2019-05-24T12:09:09.879Z/merged/00000.smoosh] with size[654,104,306].  Total size so far[29]

``

Hi Alon,

Could you post the full ingestion task log? I don’t see any issues in the snippet you posted, but there might be hints in the rest of the log as to what is happening.

The supervisor spec looks reasonable to me. There are a few fields in tuningConfig (windowPeriod and rejectionPolicy) that are not applicable for Kafka ingestion, but it shouldn’t hurt anything for them to be present.

Hi David!
thank you for your answer.
I managed to run the tasks with no failures, I just start read the data from the latest and not the earliest, maybe it caused some trouble.
But you could assist me with compaction, I manage to run compaction task, how can I open those tasks periodically on the last 3 days for example
so the compaction will fix the partition size.
thx!

Hey Alon,

Great to hear!

The coordinator process has the ability to configure periodic compaction jobs. Documentation for this functionality is described here:

http://druid.io/docs/latest/design/coordinator.html#compacting-segments
http://druid.io/docs/latest/configuration/index.html#compaction-dynamic-configuration

You would POST the configuration to /druid/coordinator/v1/config/compaction on the coordinator as described here: http://druid.io/docs/latest/operations/api-reference.html#compaction-configuration

Hope this helps!

David

Hi David,
I so appreciate you taking the time to answer my questions!
I tried the Dynamic Compaction, can you see this post please?
https://groups.google.com/forum/#!topic/druid-user/qaicRgp-Y5A

Hey Alon,

Checked out that thread and it looks like you guys figured out the ‘off-by-a-factor-of-1000’ issue :wink:

yes thx!