Ingest csv containing quoted numerical values

Hello - i’m attempting to ingest long values from quoted CSV strings (ultimately from S3 file), and then specifying long type in dimension, eg inline CSV looks like below with columns ts, name, age

example inline CSV:

"\"2022-10-01\",\"john\",\"30.5\"\n\"2022-10-02\",\"bob\",\"10.5\"\n"

however the query result showing age value as 0

ingest spec used as follows:

{
    "type": "index_parallel",
    "spec": {
      "dataSchema": {
        "dataSource": "quoted_longs",
        "dimensionsSpec": {
          "dimensions": [
            "name",
            {
              "name": "age",
              "type": "long"
            }
          ]
        },
        "timestampSpec": {
          "column": "ts",
          "format": "auto"
        },
        "metricsSpec": [],
        "granularitySpec": {
          "type": "uniform",
          "segmentGranularity": "day",
          "queryGranularity": "none",
          "intervals": [
            "2010-01-01/2030-01-01"
          ],
          "rollup": false
        }
      },
      "ioConfig": {
        "type": "index_parallel",
        "inputSource": {
          "type": "inline",
          "data": "\"2022-10-01\",\"john\",\"30.5\"\n\"2022-10-02\",\"bob\",\"10.5\"\n"
        },
        "inputFormat": {
          "type": "csv",
          "columns": [
            "ts",
            "name",
            "age"
          ]
        },
        "appendToExisting": false,
        "dropExisting": true
      },
      "tuningConfig": {
        "type": "index_parallel",
        "partitionsSpec": {
          "type": "dynamic"
        },
        "maxRowsInMemory": 25000
      }
    }
  }

Are there any special ways to define (opencsv?) parser config to convert sting long values to long

How are numerical values within quoted CSV files typically ingested?

Any help appreciated,

Thanks

Roger

Try adding a transformation that casts it to long.
Here are the docs on that.

Thanks @Sergio_Ferragut - in this instance changing dimension type from “long” to “double” resolved the issue, no transformation required :slight_smile:

1 Like

That makes even more sense. :slight_smile: