How to batch load daily segment data over small hourly segments

Hi,

We keep generated hourly raw data and store at S3 bucket. Our segmentGranularity is DAY in druid and now when we batch ingest hourly data using index task, druid considers only the last hour data. Is there any way to tell Druid to use all hourly segments?

Thanks,
Udit

Hey Udit,

When doing batch indexing, you should provide enough input files to ‘fill up’ the full segmentGranularity. So, if you’re using segmentGranularity = DAY, you should provide a full day’s worth of input files. If you just want to index an hour at a time, you can use segmentGranularity = HOUR.

Got it :slight_smile: Many thanks!!

Regards,

Udit