What cause multiple segments got created for one "segmentGranularity"

Hi, I am using druid 0.13 and kafka indexing service to ingest data. I set “segmentGranularity”: “DAY” and

“tuningConfig”: {

"type": "kafka",

"maxRowsInMemory" : 100000,

"maxBytesInMemory" : 0,

"maxRowsPerSegment" : 500000,

"maxTotalRows" : null,

"intermediatePersistPeriod" : "PT5M"

}

I’v noticed that a lot of small segments are created under each day. we only have about 3M record per day, and data size isn’t big (less than 500M ). I am expecting to create 1 segment per day. Could anyone help what I did wrong?

Screen Shot 2019-04-03 at 4.03.26 PM.png

Hi

Segments are created based on ingestion properties and sometimes it’s difficult to get just one segment created.

It’s better to enable automatic compaction ->
http://druid.io/docs/latest/configuration/index.html#compaction-dynamic-configuration

Or run Compaction jobs manually ->
http://druid.io/docs/latest/ingestion/compaction.html

Thanks & Rgds

Venkat

image001.png

Check tuning config. Specifically “targetPartitionSize”

image001.png

Hi, Thank you both for input.

  1. apparently, I used the wrong setting in tuningConfig, should be targetPartitionSize not maxRowsPerSegment. The document says the default value is 5000000. hmm, we only have 3-4M records per day. So I would assume that only 1 segment will get created per day.

Venkat, could you explain a bit more what you mean “Segments are created based on ingestion properties”

  1. I would like to enable auto compaction in any case. So I wrote a simple json

{

“dataSource”:“mts”,

“maxNumSegmentsToCompact”:50

}

but no task is created when I did

curl -XPOST -H’Content-Type: application/json’ -d @d_compact.json http://:8090/druid/coordinator/v1/config/compaction

Did I miss something?

Thank you so much for helping

Christine