doubleLast support during Hadoop Batch Ingestion

Hi Team,

We are currently trying to do a doubleLast during a 15 mins rollup granularity.

But according to the documentation doubleLast is not supported during the Batch ingestion

Below is the code snippet

@Override

public AggregateCombiner makeAggregateCombiner()

{

throw new UOE(“DoubleLastAggregatorFactory is not supported during ingestion for rollup”);

}

Error =>

Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to io.druid.collections.SerializablePair

at io.druid.query.aggregation.last.DoubleLastAggtest_dataset1.csvregatorFactory$1$1.aggregate(DoubleLastAggregatorFactory.java:118) ~[druid-processing-0.12.0.jar:0.12.0]

at io.druid.segment.incremental.OnheapIncrementalIndex.doAggregate(OnheapIncrementalIndex.java:188) ~[druid-processing-0.12.0.jar:0.12.0]

at io.druid.segment.incremental.OnheapIncrementalIndex.addToFacts(OnheapIncrementalIndex.java:126) ~[druid-processing-0.12.0.jar:0.12.0]

at io.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:504) ~[druid-processing-0.12.0.jar:0.12.0]

at io.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:498) ~[druid-processing-0.12.0.jar:0.12.0]

at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:631) ~[druid-indexing-hadoop-0.12.0.jar:0.12.0]

at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:500) ~[druid-indexing-hadoop-0.12.0.jar:0.12.0]

at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) ~[hadoop-mapreduce-client-core-2.7.3.jar:?]

at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) ~[hadoop-mapreduce-client-core-2.7.3.jar:?]

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) ~[hadoop-mapreduce-client-core-2.7.3.jar:?]

at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319) ~[hadoop-mapreduce-client-common-2.7.3.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_171]

Can you please let us know the reason why doubleLast is not supported? Also would like to know if it might be supported in the future.

Thanks and Regards,

Prabhu.

Hi Prabhu,

Did you figure out the reason for doubleLast not being supported during ingestion?

I think the necessary code just isn’t implemented yet. Aggregators need some extra extension points implemented to be able to work at ingestion time.