Ability to have JSON objects in multi-valued dimension arrays


The data that we are currently ingesting into Druid can contain an array of JSON objects that represents metadata about the specific metric. We had initially been serializing this metadata in such a way that we could use the flatten spec to create dimensions for the tags that we were explicitly knew about. This means however that we were losing “custom” tags that we aren’t aware of which may be being provided by the external service.

I have recently read the excellent article by Netsil as to time series database they looked at and why the chose Druid for DevOps monitoring of their services, see https://blog.netsil.com/a-comparison-of-time-series-databases-and-netsils-use-of-druid-db805d471206#.xtr9704tl for further information. As part of the article they refer to their usage of multi-valued dimensions for storing and indexing tags for infrastructure and container names etc. This also includes a nice screenshot showing this presented in their UI.

My question is whether or not Druid is able to support the ability to ingest an array of JSON objects and store them as a multi-valued dimension. I know that this is possible using an array of repeated string values, but that doesn’t quite handle the complexity of name/value pairs. An example of our current data would be:

“tags”: [{“name”: “hostname”, “value”: “server.hostname.internal”}, {“name”: “service”, “value”: “identity”}, {“name”: “installationId”, “value”: “1197821993”}, {“name”: “custom-tag-1”, “value”: “custom-value-a”}]

Or even restructured to:

“tags”: [{“hostname”, “server.hostname.internal”}, {“service”, “identity”}, {“installationId”, “1197821993”}, {“custom-tag-1”, “custom-value-a”}]

As I mentioned earlier, we’re able to use the flatten spec to get the “hostname”, “service”, and “installationId” tags…but not the “”… etc as we don’t know what it is called at time of creating the ingestion spec or consuming the data.

Understand if not, but would be good to know if it is possible to achieve something of this nature.