How to config the size of shard,which load data from kafka?

I use indexing service to load data from kafka.the data from kafka is not large.each day no greater 50m. so the size of the shard generate by indexing service is small.
I want to generate a segment by a day.
"segmentGranularity": "DAY"

but,a segments contain many small shards.each shard no more than 10m.
i have config the
  "tuningConfig": {
    "type": "kafka",
    "maxRowsPerSegment": 500000000
it also generate a lot small can i config it generate a big shard.

What is the average segment size you are getting?

Rommel Garcia
Director, Field Engineering

a day contains a segment, which have 48 shards.

the size of a shard is between 600KB~800KB

Hi :

would you attach one of the ingestion task logs here? it may provide us more info about why it has to keep creating new shards.

Another method is you can run post ingestion compaction task regularly to merge the small shards into more optimized ones.



but,there's another question, I druid cluster can't print any log in the integration task.
i don't know why, a task complete, and just print:
Thread-2 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started
    at io.druid.common.config.Log4jShutdown.addShutdownCallback(
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(
    at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(
    at org.apache.logging.log4j.core.LoggerContext.start(
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(
    at org.apache.logging.log4j.LogManager.getContext(
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(
    at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(
    at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(
    at org.slf4j.LoggerFactory.getLogger(
    at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(
    at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(
    at org.apache.commons.logging.LogFactory.getLog(
    at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(
    at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(
    at org.apache.hadoop.hdfs.DFSClient.close(
    at org.apache.hadoop.hdfs.DistributedFileSystem.close(
    at org.apache.hadoop.fs.FileSystem$Cache.closeAll(
    at org.apache.hadoop.fs.FileSystem$Cache$
    at org.apache.hadoop.util.ShutdownHookManager$

probably druid user does not have writing privilege to the “” , it’s configured in

thanks,I have solve it。

it's because the taskDuration, and I turn up it.