Along with segment granularity, is it possible to change query granularity as well during Compaction

TROUBLESHOOTING Google Groups
#1

Hi,

Currently I have segments with HOUR segment granularity and also HOUR query granularity. Compaction allows us to change segment granularity from HOUR to DAY. On similar lines is it possible to change the query granularity from DAY to HOUR as well during compaction? I looked at Druid compaction task configuration, and I don’t see any option as such for changing query granularity during compaction.

Regards,

Vinay

#2

HI Vinay,

Once the data is ingested with higher queryGranularity level ( let say DAY ) into the data source ( let say D1) it doesn’t make any sense to change it to lower granularity level ( HOUR) as you actal data ingested into the datasource D1 does not have hour level details.

Do for example if you input data is :

{“timestamp”:“2019-10-21T21:31:01.498Z”,“id”:“1”}

Post ingestion with " DAY level query granularity level, It will look like :

“timestamp”:“2019-10-21T00:00:00.498Z”,“id”:“1”}

If you see , its just have DAY level details here.

Hope this helps .

Thanks and Regards,

Vaibhav

#3

Hi Vinay,

I do not think we can change the query granularity once the data is ingested into Druid. Query granularity will summarize the data and then stores the data into Druid.

So if you have Day query granularity, then the data for a given day will be completely summarized into one date(timestamp) for all the events from that day.

For-ex, Timestamp 2015-12-21T10:45:00 will be 2015-12-21T00:00:00 if the query granularity is set to “Day”.

Thanks,

Hemanth

#4

Sorry, my bad. I meant can we change query granularity from HOUR to DAY (wrote vice-versa).

Regards,

Vinay

#5

Got it. Also, I don’t see an option as well to change query granularity during compaction. Looks like it is not supported then.

Regards,

Vinay

#6

HI Vinay,

I dont think the Compaction task will help in changing the queryGranularity.

The purpose of the compaction task to compacting a large number of smaller segments ( Kbs -300MB) into decent size segments ( 500MB-900MB) to improve the query performance.

Thanks,

Vaibhav

#7

I do not see the option to change query granularity in both Auto and manual compaction grammar.

https://druid.apache.org/docs/0.15.0-incubating/configuration/#compaction-dynamic-configuration

https://druid.apache.org/docs/0.15.0-incubating/ingestion/compaction.html

Thanks,

Hemanth