Enable forceExtendableShardSpecs and Kafka Indexing Service not compatible?

Hi Team,

We are using Kafka Indexing Service for real time ingestion (no time window) and a Hadoop reingest task for compacting the number of segments for the previous day.

After we faced the error

com.metamx.common.ISE: Could not allocate segment for row with timestamp


and searching in this group, we enabled in our reingest job the parameter forceExtendableShardSpecs
Now, while the realtime task run fine, the reingest task remains in the waiting task list (as it’s waiting for the real time taks lock?!)

How we can overcome this situation? Do we need to stop the supervisor to let the reingest run? Or maybe setting a time window on the supervisor spec? Either way, not seems like a reasonable solution as we need to be able to accept in real time old data and also we cannot afford to wait couple of hours for reingest to run without real time ingestion.

Thanks for your time,

Hi Dan,

There is a bug in 0.10.0 that could cause this (https://github.com/druid-io/druid/pull/4257) which is fixed in 0.10.1. You can find a release candidate here: http://druid.io/downloads.html and expect a final release soon.

Thanks Gian - funny enough, after sitting in waiting list for couple of hours, the job starts running and finally finished with success
We have multiple datasource and the issue didn’t occur on other ones

Thanks again for your time