Roll-up data

With rollup being true for data older than year,
a. Do we need to consider additonal storage for historical segments?

b. can we still get the broker to run query on those rolled up data by date or month ?

Chitra

Hi Chitra,

I’m understanding your setup as: ingesting recent data with rollup:false, and then periodically re-indexing data that is older than a year with roll-up enabled. If I’m misunderstanding your setup, please clarify so we can provide a more helpful response.

In this case:

a) When you re-index the data with roll-up enabled, the rolled-up segments will overshadow and replace the previous ‘unrolled’ segments on the historicals. Assuming that the space requirements for the data has decreased after roll-up (which it should, otherwise why do roll-up?), the net effect is that you will need less space on the historicals than you did for the unrolled segments, but you will still need enough capacity to hold all the data. If you no longer care about data after a certain period of time and don’t want that data to take up space on the historicals, you would handle this by using a load/drop retention rule.

b) You can query rolled-up data to the granularity fineness of whatever queryGranularity is set at during ingestion time. So if you want to be able to query by individual day, your queryGranularity would need to be set to DAY or finer. If you set queryGranularity to MONTH, you would not be able to get results back grouped by individual days or weeks as this information would be lost at ingestion time.