Create metric dynamically using dimension value

Hi,

Created a kafka type ingestion task with rollup as enabled.
JSON ingestion e.g.,
{“timestamp”:“2022-02-27T17:17:20Z”,“type”:“t1”,“name”:“M51”,“value”:1,“rule”:“sum”} //1
{“timestamp”:“2022-02-27T17:18:20Z”,“type”:“t1”,“name”:“M51”,“value”:2,“rule”:“sum”} //2
{“timestamp”:“2022-02-27T17:19:20Z”,“type”:“t1”,“name”:“M51”,“value”:3,“rule”:“sum”} //3

{“timestamp”:“2022-02-27T18:17:20Z”,“type”:“t1”,“name”:“M52”,“value”:1,“rule”:“min”} //4
{“timestamp”:“2022-02-27T18:18:20Z”,“type”:“t1”,“name”:“M52”,“value”:2,“rule”:“min”} //5
{“timestamp”:“2022-02-27T18:19:20Z”,“type”:“t1”,“name”:“M52”,“value”:3,“rule”:“min”} //6
{“timestamp”:“2022-02-27T18:20:20Z”,“type”:“t1”,“name”:“M51”,“value”:4,“rule”:“sum”} //7
{“timestamp”:“2022-02-27T18:21:20Z”,“type”:“t1”,“name”:“M51”,“value”:5,“rule”:“sum”} //8

{“timestamp”:“2022-02-27T19:17:20Z”,“type”:“t1”,“name”:“M53”,“value”:1,“rule”:“max”} //9
{“timestamp”:“2022-02-27T19:18:20Z”,“type”:“t1”,“name”:“M53”,“value”:2,“rule”:“max”} //10
{“timestamp”:“2022-02-27T19:19:20Z”,“type”:“t1”,“name”:“M53”,“value”:3,“rule”:“max”} //11
{“timestamp”:“2022-02-27T19:20:20Z”,“type”:“t1”,“name”:“M52”,“value”:4,“rule”:“min”} //12
{“timestamp”:“2022-02-27T19:21:20Z”,“type”:“t1”,“name”:“M52”,“value”:5,“rule”:“min”} //12

each name M51, M52 and M53 associate with any one of aggregation function ( sum, min, max )

currently i am using metricSpec as

“metricsSpec”: [
{
“type”: “count”,
“name”: “row_count”
},
{
“type”: “longSum”,
“name”: “sum_value”,
“fieldName”: “value”
},
{
“type”: “longMin”,
“name”: “min_value”,
“fieldName”: “value”
},
{
“type”: “longMax”,
“name”: “max_value”,
“fieldName”: “value”
}
],

But i don’t want sum, min or max values to be stored for all rows. instead need single column to store aggregated value.

Now i want to create metric dynamically ,
for 1,2,3 ; rule dimension value is sum, so i want to create longSum metric, don’t want longMin or longMax values to be stored in datasource
{ “type” : “longSum”, “name” : “sum_metric”, “fieldName” : “value” }
next 4,5,6 rule dimension value is min , so i want to create longMin metric , don’t want longSum or longMax values to be stored in datasource
{ “type” : “longSum”, “name” : “min_metric”, “fieldName” : “value” }
in same 1hr interval time next 7,8 rule dimension value is sum, so i want to create longSum metric, don’t want longMin or longMax values to be stored in datasource
{ “type” : “longSum”, “name” : “min_metric”, “fieldName” : “value” }

likewise there are so many data points are received,

current table schema

Expected table schema

is there any way to do this.

I haven’t attempted this. But one idea is to use “type”,“name”,“rule” as dimensions, enable rollup, and use a javascript aggregate UDF to execute either min, max or sum based on the “rule” parameter.

Have a look at these docs for more on Javascript functionality.

I think it should be doable. Let us know how it goes.

I’m curious, can you share the use case for this?

AH!! I love these kinds of questions @Navin thanks for asking… but I can’t promise I have the answer (!!!)

I wondered whether you could use a filtered aggregator somehow… but I think you may end up with the three columns …