Druid ingestion tasks are failing

Hi,

The ingestion tasks are not running. After zk cluster formation failed, we started noticing the following error even after zk cluster formation recovered.

2019-12-29 02:08:58 ERROR [KafkaSupervisor-flowlogs] org.apache.druid.emitter.t2.T2Emitter - unknown event type [class org.apache.druid.java.util.emitter.service.AlertEvent]

2019-12-29 02:09:23 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

2019-12-29 02:09:28 INFO [KafkaSupervisor-flowlogs] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - [flowlogs] supervisor is running.

2019-12-29 02:09:28 INFO [KafkaSupervisor-flowlogs] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Creating new task group [0] for partitions [0, 32, 2, 34, 4, 36, 6, 38, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30]

2019-12-29 02:09:28 ERROR [KafkaSupervisor-flowlogs] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - SeekableStreamSupervisor[flowlogs] failed to handle notice: {class=org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor, exceptionType=class org.apache.druid.java.util.common.IAE, exceptionMessage=Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers, noticeClass=RunNotice}

org.apache.druid.java.util.common.IAE: Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers

    at org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers.minus(SeekableStreamEndSequenceNumbers.java:159) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at org.apache.druid.indexing.seekablestream.SeekableStreamDataSourceMetadata.minus(SeekableStreamDataSourceMetadata.java:95) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.resetInternal(SeekableStreamSupervisor.java:1147) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.getOffsetFromStorageForPartition(SeekableStreamSupervisor.java:2380) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.generateStartingSequencesForPartitionGroup(SeekableStreamSupervisor.java:2357) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.createNewTasks(SeekableStreamSupervisor.java:2254) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1013) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:265) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:724) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_162]

    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]

    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]

2019-12-29 02:09:28 ERROR [KafkaSupervisor-flowlogs] org.apache.druid.emitter.t2.T2Emitter - unknown event type [class org.apache.druid.java.util.emitter.service.AlertEvent]

Thanks,

Dhiman

HI,

Can someone explain what conditions can trigger the following errors ?

  • Creating new task group [0] for partitions [0, 32, 2, 34, 4, 36, 6, 38, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30]

2019-12-29 16:27:52 INFO [KafkaSupervisor-flowlogs] org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Discovered coordinator kafka-broker:9092 (id: 2147483645 rack: null) for group kafka-supervisor-klphljkm.

2019-12-29 16:27:52 ERROR [KafkaSupervisor-flowlogs] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - SeekableStreamSupervisor[flowlogs] failed to handle notice: {class=org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor, exceptionType=class org.apache.druid.java.util.common.IAE, exceptionMessage=Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers, noticeClass=RunNotice}

org.apache.druid.java.util.common.IAE: Expected instance of org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers, got org.apache.druid.indexing.seekablestream.SeekableStreamStartSequenceNumbers

at org.apache.druid.indexing.seekablestream.SeekableStreamEndSequenceNumbers.minus(SeekableStreamEndSequenceNumbers.java:159) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

at org.apache.druid.indexing.seekablestream.SeekableStreamDataSourceMetadata.minus(SeekableStreamDataSourceMetadata.java:95) ~[druid-indexing-service-0.14.0-incubating.jar:0.14.0-incubating]

Dhiman

Hey Dhiman,

I replied in your other thread, https://groups.google.com/forum/#!topic/druid-user/kthC3kPKeOE.

Gian