JavaScript sample for parsing json array data using kafka ingestion spec

Hi,
In kafka topic data is in json array format like below,

{

“parking_data”: [

{

  "parkingId": 15,

  "parkingName": "SIVAGNANAM STREET",

  "latitude": "13.039619470005132",

  "longitude": "80.23578990250824",

  "location": "T.Nagar",

  "2WCapacity": "120",

  "4WCapacity": "40",

  "2WOccupied": "0",

  "4WOccupied": "0",

  "dateTime": "31-07-2019 10:36:01",

  "zoneId": "1",

  "RoadName": "Test",

  "WardNO": "01",

  "AreaName": "SIVAGNANAM STREET"

},

{

  "parkingId": 20,

  "parkingName": "GATE1",

  "latitude": "13.0834552",

  "longitude": "80.2715324",

  "location": "Amma Maligai ",

  "2WCapacity": "0",

  "4WCapacity": "40",

  "2WOccupied": "0",

  "4WOccupied": "0",

  "dateTime": "31-07-2019 10:36:01",

  "zoneId": "1",

  "RoadName": "Test",

  "WardNO": "01",

  "AreaName": "Ripon building"

}]}

Anybody have any idea how to create kafka ingestion spec for parsing the dynamic json array and insert into druid.Please suggest me and let me know how to do using druid 0.10.Any help is appreciated.

Prasanna.P

You will need to create a flatten spec - similar to the following. You can also use jq.

https://druid.apache.org/docs/latest/ingestion/flatten-json

{

“type”: “kafka”,

“dataSchema”: {

“dataSource”: “jnpr.jvision-5dad”,

“parser”: {

“type”: “string”,

“parseSpec”: {

“format”: “json”,

“flattenSpec”: {

“useFieldDiscovery”: true,

“fields”: [{

“type”: “path”,

“name”: “Optics_diag.optics_diag_stats.optics_lane_diag_stats.lane_number”,

“expr”: “$[‘key_fields’][‘Optics_diag.optics_diag_stats.optics_lane_diag_stats.lane_number’]”

},

{

“type”: “path”,

“name”: “Optics_diag.if_name”,

“expr”: “$[‘key_fields’][‘Optics_diag.if_name’]”

},

{

“type”: “path”,

“name”: “edges.class_stats.priority”,

“expr”: “$[‘key_fields’][‘edges.class_stats.priority’]”

},

{

“type”: “path”,

“name”: “edges.source_type”,

“expr”: “$[‘key_fields’][‘edges.source_type’]”

},

Eric Graham

Solutions Engineer -** **Imply

**cell: **303-589-4581

email: eric.graham@imply.io

www.imply.io

Thank You Eric for the reply. I tried json flatten spec like below, but its not parsing the data. Can you suggest me ,how to solve this.
spec:

“parser”: {

“type”: “string”,

“parseSpec”: {

“format”: “json”,

“flattenSpec”: {

“useFieldDiscovery”: true,

“fields”: [{

“type”: “path”,

“name”: “parkingId”,

“expr”: “$[‘parking_data’][‘parking_data.parkingId’]”

},

{

“type”: “path”,

“name”: “parkingName”,

“expr”: “$[‘parking_data’][‘parking_data.parkingName’]”

}

]

},

Prasanna.P

I think it should be a format similar to this…

“fields”: [{

“type”: “path”,

“name”: “parkingId”,

“expr”: “$[‘parking_data’][‘parkingId’]”

Eric Graham

Solutions Engineer -** **Imply

**cell: **303-589-4581

email: eric.graham@imply.io

www.imply.io