Druid Ingestion

Hi All,

I am trying to ingest data in druid from kafka… This is how data looks like in kafka.

{“Machine”:“CMGG-012”,“Time_Stamp”:“1522233364000”,“Z2Temp”:463020.0,“XTemp”:630.0,“YTemp”:23370.0,“ZTemp”:30.0,“ATemp”:101580.0,“CTemp”:251010.0}

I am taking timestamp in milliseconds as string … It will work or I have to convert it into long int?

Hi Prakhar,
String should work, you will need to set your timestamp format to “millis”.

Let us know, In case you face any issues.

Hi Nishant,

This is my ingestion schema…

{

“type”: “kafka”,

“dataSchema”: {

“dataSource”: "testMemReg,

“parser”: {

“type”: “string”,

“parseSpec”: {

“timestampSpec”: { “column”: “Time Stamp”, “format”: “millis” },

“dimensionsSpec”: {

“dimensions”: [“Machine”,“Time Stamp”,“Z2 - Temp”,“X - Temp”]

},

“format”: “json”

}

},

“metricsSpec”: [

{ “type”: “count”, “name”: “count” }

],

“granularitySpec”: {

“type”: “uniform”,

“segmentGranularity”: “hour”,

“queryGranularity”: “none”

}

},

“tuningConfig”: {

“type”: “kafka”,

“maxRowsPerSegment”: 5000000

},

“ioConfig”: {

“topic”: “MemRegDataV2”,

“consumerProperties”: {

“bootstrap.servers”: “localhost:6667”

},

“taskCount”: 1,

“replicas”: 1,

“taskDuration”: “PT60S”

}

}

and log file attached. It gives exception after success.

log.txt (76.7 KB)

The exception is related to publising of metrics, probably your Ambari Metrics Server is not running or is not configured properly.
This should not affect indexing and you should be able to query the data successfully.

2018-05-13T05:16:59,764 INFO [AmbariMetricsEmitter-0] io.druid.emitter.ambari.metrics.AmbariMetricsEmitter - Unable to connect to collector, [http://localhost.localdomain:6188/ws/v1/timeline/metrics](http://localhost.localdomain:6188/ws/v1/timeline/metrics)