We are using Kafka Indexing Service for real time ingestion (no time window) and a Hadoop reingest task for compacting the number of segments for the previous day.
After we faced the error
com.metamx.common.ISE: Could not allocate segment for row with timestamp
and searching in this group, we enabled in our reingest job the parameter forceExtendableShardSpecs
Now, while the realtime task run fine, the reingest task remains in the waiting task list (as it’s waiting for the real time taks lock?!)
How we can overcome this situation? Do we need to stop the supervisor to let the reingest run? Or maybe setting a time window on the supervisor spec? Either way, not seems like a reasonable solution as we need to be able to accept in real time old data and also we cannot afford to wait couple of hours for reingest to run without real time ingestion.
Thanks for your time,