[druid-user] Kafka indestion failing when done with supervisor task with exception : Could not allocate segment

Hi Rajnandini,

I have seen a similar error when I have run a hadoop compaction job and then receive an event in the compacted timeframe. Something about how the hadoop indexing task wrote out the segments made it impossible for the kafka indexer to append extra data (probably something to do with partitioning). I worked around the issue by configuring my supervisor task to ignore data from 2 weeks ago, and only compact data older than 2 weeks. I have no idea whether there are other situations in which this error presents, but hopefully this helps you out.

Regards,

–Ben

Hi Ben,
Yes, you are correct, batch job by default generates non-extendable shard specs which can lead to this error.

We have introduced a new flag in 0.9.2, forceExtendableShardSpecs which can be set to true to avoid this,
http://druid.io/docs/0.9.2-rc3/ingestion/batch-ingestion.html