Flatten JSON with Tranquility doesn't work

Hi,

I’m ingesting json data and giving FlattenSpec in tranquility config file. Below is tranquility config file.

{
  "dataSources" : [
    {
      "spec" : {
        "dataSchema" : {
          "dataSource" : "ds_test",
          "metricsSpec" : [
            {
              "type" : "count",
              "name" : "count"
            }
          ],
          "granularitySpec" : {
            "segmentGranularity" : "hour",
            "queryGranularity" : "none",
            "type" : "uniform"
          },
          "parser" : {
            "type" : "string",
            "parseSpec": {
              "format": "json",
              "flattenSpec": {
                "useFieldDiscovery": true,
                "fields": [
                  "dim2",
                  {
                    "type": "path",
                    "name": "foo.bar",
                    "expr": "$.foo.bar"
                  },
                  {
                    "type": "path",
                    "name": "path-metric",
                    "expr": "$.nestmet.val"
                  },
                  {
                    "type": "path",
                    "name": "hello-0",
                    "expr": "$.hello[0]"
                  },
                  {
                    "type": "path",
                    "name": "hello-4",
                    "expr": "$.hello[4]"
                  },
                  {
                    "type": "path",
                    "name": "world-hey",
                    "expr": "$.world[0].hey"
                  },
                  {
                    "type": "path",
                    "name": "worldtree",
                    "expr": "$.world[1].tree"
                  },
                  {
                    "type": "path",
                    "name": "first-food",
                    "expr": "$.thing.food[0]"
                  },
                  {
                    "type": "path",
                    "name": "second-food",
                    "expr": "$.thing.food[1]"
                  }
                ]
              },
              "dimensionsSpec" : {
                "dimensions" : ["dim2","first-food", "second-food"],
                "dimensionsExclusions": ["ignore_me"]
              },
              "timestampSpec" : {
                "format" : "auto",
                "column" : "timestamp"
              }
            }
          }
        },
        "tuningConfig" : {
          "type" : "realtime",
          "windowPeriod" : "PT10M",
          "intermediatePersistPeriod" : "PT10M",
          "maxRowsInMemory" : 75000
        }
      },
      "properties" : {
        "task.partitions" : "1",
        "task.replicants" : "1"
      }
    }
  ],
  "properties" : {
    "zookeeper.connect" : "localhost",
    "http.port" : "8200",
    "http.threads" : "6"
  }
}

``

The above configuration is an example from http://druid.io/docs/latest/ingestion/flatten-json.html . After applying the configuration, tranquility is restarted. And I’m ingesting below data

{

“timestamp”: “2017-12-24T04:10:53.155Z”,

“dim1”: “qwerty”,

“dim2”: “asdf”,

“dim3”: “zxcv”,

“ignore_me”: “ignore this”,

“metrica”: 9999,

“foo”: {“bar”: “abc”},

“foo.bar”: “def”,

“nestmet”: {“val”: 42},

“hello”: [1.0, 2.0, 3.0, 4.0, 5.0],

“mixarray”: [1.0, 2.0, 3.0, 4.0, {“last”: 5}],

“world”: [{“hey”: “there”}, {“tree”: “apple”}],

“thing”: {“food”: [“sandwich”, “pizza”]}

}

``

After ingestion, tranquility logs show the data source schema. But the schema doesn’t show flatten spec.

{

“type” : “index_realtime”,

“id” : “index_realtime_ds_test_2017-12-24T04:00:00.000Z_0_0”,

“resource” : {

“availabilityGroup” : “ds_test-2017-12-24T04:00:00.000Z-0000”,

“requiredCapacity” : 1

},

“spec” : {

“dataSchema” : {

“dataSource” : “ds_test”,

“parser” : {

“type” : “map”,

“parseSpec” : {

“format” : “json”,

“timestampSpec” : {

“column” : “timestamp”,

“format” : “millis”,

“missingValue” : null

},

“dimensionsSpec” : {

“dimensions” : [ “dim2”, “first-food”, “second-food” ],

“spatialDimensions” :

}

}

},

“metricsSpec” : [ {

“type” : “count”,

“name” : “count”

} ],

“granularitySpec” : {

“type” : “uniform”,

“segmentGranularity” : “HOUR”,

“queryGranularity” : {

“type” : “none”

},

“rollup” : true

}

},

“ioConfig” : {

“type” : “realtime”,

“plumber” : null,

“firehose” : {

“type” : “clipped”,

“interval” : “2017-12-24T04:00:00.000Z/2017-12-24T05:00:00.000Z”,

“delegate” : {

“type” : “timed”,

“shutoffTime” : “2017-12-24T05:15:00.000Z”,

“delegate” : {

“type” : “receiver”,

“serviceName” : “firehose:druid:overlord:ds_test-004-0000-0000”,

“bufferSize” : 100000

}

}

}

},

“tuningConfig” : {

“shardSpec” : {

“type” : “linear”,

“partitionNum” : 0

},

“rejectionPolicy” : {

“type” : “none”

},

“buildV9Directly” : false,

“maxPendingPersists” : 0,

“intermediatePersistPeriod” : “PT10M”,

“windowPeriod” : “PT10M”,

“type” : “realtime”,

“maxRowsInMemory” : 75000

}

}

}

``

The output of the druid select endpoint shows below data.

[

{

“timestamp”: “2017-12-24T00:00:00.000Z”,

“result”: {

“pagingIdentifiers”: {

“ds_test_2017-12-24T03:00:00.000Z_2017-12-24T04:00:00.000Z_2017-12-24T03:57:07.254Z”: 0

},

“dimensions”: [

“dim2”,

“first-food”,

“second-food”

],

“metrics”: [

“count”

],

“events”: [

{

“segmentId”: “ds_test_2017-12-24T03:00:00.000Z_2017-12-24T04:00:00.000Z_2017-12-24T03:57:07.254Z”,

“offset”: 0,

“event”: {

“timestamp”: “2017-12-24T03:10:53.155Z”,

“dim2”: “asdf”,

“first-food”: null,

“second-food”: null,

“count”: 1

}

}

]

}

}

]

``

I was expecting first-food, second-food dimensions to show some values instead of nulls. Am I doing something wrong here?

I’m using tranquility 0.8.2 and druid 0.10.0