Re: [druid-user] Apache Druid – Ingesting multiple objects in JSON returns only single row

Hey Wilhelm,
I think that essentially you’re looking for something like “explode” functionality, so for 1 input row, you want N output rows, correct?
Assuming that’s the case, I don’t think Druid supports such functionality ATM (see https://github.com/apache/druid/pull/8698).
I think you’ll have to pre-process the data before ingesting it to Druid.

Hey Wilhelm,
After reading your last email a few times, I must admit I’m still not sure if my comment re/ “explode” functionality was relevant or now, so let me make sure I understand correctly:

  1. When Druid retrieves a record from your HTTP server, that record contains multiple JSON objects (whether it’s 3 or 30), correct?
    An example for such a single record would be (I intentionally removed white spaces):
    {“1”:{“lastseen”:“2021-02-15T05:02Z”,“name”:“Temperature Sensor A”,“state”:{“alert”:“none”,“level”:152,“on”:true,“reachable”:true}},“3”:{“lastseen”:“2021-02-15T05:01Z”,“name”:“Temperature Sensor B”,“state”:{“alert”:“none”,“level”:13,“on”:false,“reachable”:true}},“4”:{“lastseen”:“2021-02-15T05:00Z”,“name”:“Temperature Sensor Backup”,“state”:{“alert”:“none”,“level”:76,“on”:true,“reachable”:true}}}
  2. If so - then essentially you are trying to produce N (again, whether it’s 3 or 30) output records for 1 input record, right?
    So based on the previous record, you’d like Druid to produce 3 records, e.g:
    lastseen | name
    2021-02-15T05:02Z | Temperature Sensor A
    2021-02-15T05:01Z | Temperature Sensor B
    2021-02-15T05:00Z | Temperature Sensor Backup

If the answer to both questions is “yes”, then unfortunately I’m not familiar with a way to achieve that without pre-processing the data, but I’ll be happy to be proven wrong here :slight_smile:

Thanks and good luck!
Itai

+1 Itai here, Willhelm - Druid currently collects one event (row) from your source and then outputs one event (row). Likely you will want something to extract each row and emit it to a stream that Druid watches.