Optimal values for windowPeriod when using "minute" or "second" segment granularity

Druid supports segment granularity as low as “minute” and “second”. This would come in handy when the use case demands such low granularity. However for such low granularity what should be the optimal value for the properties: windowPeriod and *intermediatePersistPeriod? *

Apart from these properties there is a default firehoseGracePeriod of 5 minutes, which means a real-time task created (via tranquility) given the segment granularity of “minute” or “second” will run for 5 minutes over the windowPeriod.

So what should be the optimal values for windowPeriod and intermediatePersistPeriod while setting the segment granularity of “minute”. It wouldn’t make much sense to keep the task running for 15 minutes ( given that only finite number of real-time tasks can run at a time)

Thanks,

Prathamesh

Segment granularities of “minute” are technically supported but they are a bit niche. Are you sure you need them? Maybe you would be good with a more typical segment granularity like “hour” and having a queryGranularity of “minute”?

Hi Gian,

We are currently using segment Granularity of “fifteen_minute”. This would pretty much suffice for most practical purposes.

I was just wondering what would be the implications when one uses “minute” or “second” granularity which is in fact supported by druid specifically from the point of setting windowPeriod and intermediatePersistPeriod greater than granularity. For most cases granularity of “second” would not be needed but one might use “minute” if the use case demands (although i agree it won’t be common), and in that case there might be problems since real-time tasks would be spawned for every minute and they will keep running for, say 10 minutes(windowPeriod+ ***firehoseGracePeriod ). ***

I was curious as to what would be the ideal values for above parameters (windowPeriod & intermediatePersistPeriod) for “minute” and “second” (“second” being more puzzling)

Thanks,

Prathamesh

Well, this setup would not be common. But if you did have segment granularities that short, you’d probably want to set windowPeriod relatively short too, to avoid needing too many tasks to run at once. It doesn’t really matter much what you set intermediatePersistPeriod too, since the task won’t wait for it.

Thanks Gian. I think for “minute” granularity, one should keep windowPeriod close to a minute or two. Not sure what would be a suitable value for “second” granularity though. May be windowPeriod of 1 minute would work.