Druid error : Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers,

Hello Druid Team

We are using Apache Druid 0.15.1 as our Analytics Engine with Kafka Ingestion.

We are seeing this error:

“Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers,”

intermittently in our stack on docker swarm. The error comes up when one or more of containers go down and come back up, middlemanager, coordinator, or kafka. The exact sequence of steps leading to this issue has not been determined yet.

I read from Apache Druid GitHub issues that one solution is to upgrade to Apache Druid 0.16.0. This is difficult choice for us in our late stage of testing towards a release.

What are our other options? Please let us know.

Stack Trace :

org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - SeekableStreamSupervisor[apim_metrics] failed to handle notice: {class=org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor, exceptionType=class org.apache.druid.java.util.common.IAE, exceptionMessage=Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers, noticeClass=RunNotice} org.apache.druid.java.util.common.IAE: Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers at org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers.minus(SeekableStreamEndSequenceNumbers.java:159) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.SeekableStreamDataSourceMetadata.minus(SeekableStreamDataSourceMetadata.java:95) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.resetInternal(SeekableStreamSupervisor.java:1171) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.getOffsetFromStorageForPartition(SeekableStreamSupervisor.java:2466) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.generateStartingSequencesForPartitionGroup(SeekableStreamSupervisor.java:2443) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.createNewTasks(SeekableStreamSupervisor.java:2341) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1037) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:290) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:748) ~[druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

Thanks

Pushpa

Hi Pushpa,
By any chance , did you have any gap or suspension of your kafka consumer job because of your docker containers going down?

Do you suspect because of that gap in reading, some data in kafka topic might have been dropped due to topic retention rules(default might be 24 hours) and your kafka consumer might bw looking for that unavailable data ?

If that’s the case, you might have to reset your supervisor.

Thank you.

–siva