Do we have to set rollup=false to disable rollup for kafka indexing?

By default, rollup=true is set for batch indexing. Therefore we need to set rollup=false to disable it.

How about kafka indexing? is it enable by default and if yes, can we disable it?

yes, you can specify rollup=false as part of granularitySpec for kafka indexing service also.

Note that this only works in 0.9.2+

Hi guys,

I have two questions:

  1. So it will not work even if we set rollup=false for kafka indexing service on druid 0.9.2, it will always roll-up rows?

  2. About the kafka indexing service roll-up feature: I know “across different taskDurations (tasks), new task will create new segment vertically without modifying old segments created by old tasks if these segments span same granularity time period.” Then if one row A is generated in previous taskDuration, which is already pushed to deep storage and a new row B (should roll-up with row A) is generated inside this taskDuration, will kafka indexing service roll-up them? One row is published and another is delayed into current taskDuration.

Best wishes,

By Linbo

Hey Linbo,

  1. If you set rollup to false on druid 0.9.2+ then the Kafka indexing service (or any other indexing method) won’t roll up rows.

  2. Yeah, if that happens, the rows will not be rolled up after stream ingestion. If you want to roll them up, you could do it by running a batch reindexing job on the same interval. In general, Druid streaming ingestion methods don’t guarantee perfect rollup at ingestion time. This is fine if you view rollup as a space saving optimization, since it will be applied at query time too.

Hey Gian,

“it will be applied at query time too” - just want to clarify that the rollup at query time uses the aggregation specified by the original ingestion spec’s metricsSpec attribute? And not the aggregation specified by the query?

Regards,

Jason

It uses the aggregation specified by the query, although it only “works right” if you specify the same aggregation that was used at ingestion time. By “works right” I mean acts just as if things were fully rolled up.