Druid 0.10.0 Kafka index service (Supervisor), Number of shards diffs, reindexing task trouble

Hi!

We are using Kafka Indexing Service (KafkaSupervisor) to pull data from existing Kafka topic.
The Kafka topic seems to have 6 partitions as you can see in the following log.
2017-10-13T08:24:57,265 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver - Restored metadata[null].
2017-10-13T08:24:57,268 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[0] to offset[19,397,761].
2017-10-13T08:24:57,268 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[1] to offset[20,443,807].
2017-10-13T08:24:57,268 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[2] to offset[20,451,449].
2017-10-13T08:24:57,268 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[3] to offset[24,643,150].
2017-10-13T08:24:57,268 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[4] to offset[22,543,016].
2017-10-13T08:24:57,268 INFO [task-runner-0-priority-0] io.druid.indexing.kafka.KafkaIndexTask - Seeking partition[5] to offset[26,742,452].
2017-10-13T08:24:57,276 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
Granularity is set to HOUR and task is running every hour, without (at the moment) replication.
“granularitySpec”: {
“type”: “uniform”,
“segmentGranularity”: “HOUR”,
“queryGranularity”: “MINUTE”
}
“ioConfig”: {
“topic”: “my_kafka_topic_with_json”,
“consumerProperties”: {
“bootstrap.servers”: “hadoop11x:9092,hadoop11x:9092,hadoop11x.at:9092”
},
“taskCount”: 1,
“replicas”: 1,
“taskDuration”: “PT1H”
}
I expected to have 6 shards per hour, but mostly getting 12 and just sometimes 6. (as you can see in the screenshot)
So…
a) Why are there mostly 12 shards per hour?
b) Why are there sometimes only 6 shards? Is there a data loss? (We could not see some in our charts)
c) I run a reindex task (per hand in TEST env, segment granularity DAY) every morning to get rid of all the small files which slow down querying.
This is a painfull task, because reindexing often fails. The reason seems to be wrong metadata. E.g. metadata says
there are 12 shades, but in HDFS there is just data for six shades which results in exceptions like the following.
2017-10-13T11:19:48,629 ERROR [task-runner-0-priority-0] io.druid.segment.realtime.firehose.ReplayableFirehoseFactory - Delegate firehose threw an exception, retrying (2 of 3)
java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.segment.loading.LocalLoadSpec] value failed: [/druid/segments/analytics_raw_tracking_supervisor**/2017-10-11T07:00:00.000Z_2017-10-11T08:00:00.000Z/2017-10-11T07:00:00.038Z/3/index.zip] does not exist**
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-server-0.10.0.jar:0.10.0]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:195) ~[druid-server-0.10.0.jar:0.10.0]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:154) ~[druid-server-0.10.0.jar:0.10.0]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:130) ~[druid-server-0.10.0.jar:0.10.0]
at io.druid.indexing.common.TaskToolbox.fetchSegments(TaskToolbox.java:203) ~[druid-indexing-service-0.10.0.jar:0.10.0]

Regards,
Alex