Hi, I am using druid 0.13 and kafka indexing service to ingest data. I set “segmentGranularity”: “DAY” and
"maxRowsInMemory" : 100000,
"maxBytesInMemory" : 0,
"maxRowsPerSegment" : 500000,
"maxTotalRows" : null,
"intermediatePersistPeriod" : "PT5M"
I’v noticed that a lot of small segments are created under each day. we only have about 3M record per day, and data size isn’t big (less than 500M ). I am expecting to create 1 segment per day. Could anyone help what I did wrong?
Segments are created based on ingestion properties and sometimes it’s difficult to get just one segment created.
It’s better to enable automatic compaction ->
Or run Compaction jobs manually ->
Thanks & Rgds
Check tuning config. Specifically “targetPartitionSize”
Hi, Thank you both for input.
- apparently, I used the wrong setting in tuningConfig, should be targetPartitionSize not maxRowsPerSegment. The document says the default value is 5000000. hmm, we only have 3-4M records per day. So I would assume that only 1 segment will get created per day.
Venkat, could you explain a bit more what you mean “Segments are created based on ingestion properties”
- I would like to enable auto compaction in any case. So I wrote a simple json
but no task is created when I did
curl -XPOST -H’Content-Type: application/json’ -d @d_compact.json http://:8090/druid/coordinator/v1/config/compaction
Did I miss something?
Thank you so much for helping